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( _ ^  Professional  Review  Organizations  (pros)  contract  with  the  Medicare 

program  to  review  the  necessity,  appropriateness,  and  quality  of  inpa¬ 
tient  hospital  services  received  by  the  program’s  beneficiaries.  From 
February  thr  ough  July  1986,  the  Department  of  Health  and  Human  Ser¬ 
vices’  (HHS’s)  Health  Care  Financing  Administration  (hcfa)  evaluated  the 
pros’  performance  under  their  contracts  to  determine  whether  the  con¬ 
tracts,  awarded  for  a  2-year  period,  should  be  renewed  on  a  noncompeti¬ 
tive  basis  or  whether  a  competition  should  be  held.  As  a  result  of  the 
evaluations,  hcfa  was  requiring  competition  for  about  half  of  the  pro 
contracts. 

Based  on  a  concern  that  the  high  nonrenewal  rate  indicated  problenas 
either  with  some  pros  or  with  hcfa’s  management  of  the  program,  the 
Chairman  and  Ranking  Minority  Member  of  the  Subcommittee  on 
Health,  Senate  Committee  on  Finance,  requested  that  gao  assess  (1) 
hcfa’s  pro  evaluation  methodology  and  (2)  the  adequacy  of  hcfa’s  rou¬ 
tine  monitoring  of  the  pros  for  identifying  performance  problems.  In 
addition,  gao  was  asked  to  determine  how  hcfa  decided  on  the  funding 
level  for  the  new  contracts.  ^ _ 


The  Tax  Elquity  and  Fiscal  Responsibility  Act  of  1982  replaced  Medi¬ 
care’s  utilization  review  program  with  the  pro  program.  While  the  pred¬ 
ecessor  review  organizations  were  funded  through  annual  grants,  pros 
operate  under  fixed-price  2-year  contracts  that  can  be  renewed  at  hcfa’s 
option  for  additional  2-year  periods. 

The  Social  Security  Amendments  of  1983  established  Medicare’s  pro¬ 
spective  payment  system  (pps)  and  required  each  hospital,  as  a  condition 
for  Medicare  payment,  to  have  an  agreement  with  the  pro  covering  its 
area  to  review  the  quality,  necessity,  and  appropriateness  of  care  pro¬ 
vided  to  Medicare  beneficiaries.  This  law  also  included  specific  require¬ 
ments  for  PRO  review  that  were  designed  to  assure  provision  of  quality 
care  and  protect  Medicare  from  paying  for  unnecessary  care,  pro  con¬ 
tracts  included  quality  and  cost  control  provisions  required  by  law. 

To  decide  whether  to  renew  individual  contracts,  hcfa  evaluated  the 
pro’s  performance  against  the  contract  requirements,  hcfa  asked  pros  to 
complete  a  detailed  report  covering  their  performance  during  the  first 
16  to  17  months  of  their  contracts,  and  hcfa’s  regional  offices  reviewed 
these  reports  for  accuracy.  Data  from  these  reports,  as  well  as  from 
hcfa’s  ongoing  pro  monitoring  and  from  a  contract  for  monitoring  pros 
(the  so-called  SuperPRO  contract),  were  used  by  evaluation  panels  to 
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assess  pro  performance.  The  panels  evaluated  Ih  elements  i-ombiiuHl 
into  three  areas.  To  earn  a  panel  reeommendatHMi  *■ ..  lumcompetuive 
renewal,  the  pro  had  to  pass  all  three  areas. 

GAO  reviewed  selected  aspects  of  the  50  pro  evaluations  »iA(*  alsti 
reviewed  six  evaluations  in  detail,  includinn  those  of  tw<i  i*kos  that  were 
noncompetitively  renewed  and  four  prjw  that  had  to  compete  for 
renewal,  gao’s  review  included  an  assessment  of  the  internal  contmls 
HCFA  used  for  assuring  that  the  evaluations  wert*  consistent,  fair,  and 
accurate. 


Results  in  Brief 


The  evaluation  process  had  instructions  that  were  inconsistent,  incor¬ 
rect,  or  not  properly  implemented  by  the  panels,  and  documentation  of 
the  panels*  and  hcfa’s  decisions  was  not  always  adequate,  hcka’s  con¬ 
trols  over  the  process  were  not  sufficient  to  identify  and  correct  these 
problems.  Although  gao  did  not  identify  any  instance  where  a  clearly 
wrong  renewal  decision  resulted,  the  potential  for  erroneous  decisions 
existed  because  of  the  absence  of  appropriate  controls,  hcfa  needs  to 
establish  a  better  system  of  internal  controls  over  its  future  evaluations 
of  PRO  contracts. 

The  substantial  number  of  pros  that  failed  the  evaluations  and  iicfa's 
unawareness  of  the  extent  of  the  problems  identified  through  the  evalu¬ 
ation  process  showed  that  hcfa’s  routine  monitoring  had  not  identified 
and  corrected  pro  performance  problems.  This,  in  turn,  meant  that  Medi¬ 
care  and  its  beneficiaries  may  not  have  been  receiving  all  of  the  protec¬ 
tion  intended  under  the  program,  hcfa  recognized  this  monitoring 
problem  during  the  evaluation  process  and  acted  to  strengthen  routine 
monitoring  of  the  new  pro  contracts.  Monitoring  could  be  further 
strengthened  by  giving  monitoring  personnel  more  specific  criteria  for 
measuring  pro  performance. 

HCFA  designed  the  scope  of  work  to  keep  the  cost  of  the  program  at  the 
minimum  funding  level  permitted  by  law.  gao  believes  that  hcfa  should 
design  the  scope  of  work  to  provide  reasonable  assurance  that  Medicare 
is  not  paying  for  unnecessary  care  and  that  Medicare  beneficiaries  are 
receiving  good  quality  care  and  then  determine  the  appropriate  costs. 
Also,  GAO  found  many  uncertainties  in  hcfa's  meth(xlology  for  estimat¬ 
ing  costs  to  individual  pros  in  carrying  out  the  work  required  by  their 
contracts.  As  a  result,  gao  could  not  determine  whether  individual  pros 
were  appropriately  funded. 
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’u^equate  Internal  in  several  instances,  hcfa’s  internal  control  procedures  were  inadequate 

(JOntrols  Over  Evaluations  ^  prevent  (l)  inconsistencies  in  the  evaluation  process,  (2)  improper 

application  of  the  instructions  by  the  panels,  and  (3)  inadequate  docu¬ 
mentation  of  evaluation  results.  For  example,  the  instructions  to  the 
panels  regarding  the  scoring  of  the  element  relating  to  sanctions  were 
incorrect.  Although  hcfa  officials  were  aware  of  this  problem,  their 
internal  controls  were  inadequate  to  prevent  one  of  the  six  panels 
reviewed  from  using  the  incorrect  instructions  and  as  a  result  recom¬ 
mending  competitive  contract  renewal.  This  error  was  not  found  and 
corrected  by  hcfa  until  the  pro  appealed  hcfa’s  decision  to  open  its  con¬ 
tract  to  competition,  hcfa’s  internal  controls  were  also  inadequate  to 
prevent  16  panels  from  assigning  scores  inconsistent  with  instructions 
for  one  or  more  evaluation  elements. 

In  two  instances,  hcfa’s  internal  controls  were  inadequate  to  assure  that 
the  results  of  evaluations  were  adequately  documented.  In  one  case, 
there  was  no  documentation  of  a  decision  by  hcfa  officials  to  offer  a  pro 
noncompetitive  contract  renewal  when  the  panel  had  not  recommended 
this  action. 


Five  out  of  the  six  pros  whose  evaluations  gao  examined  in  detail  had 
performance  problems  not  detected  by  hcfa  program  monitors  until  the 
renewal  evaluation,  which  took  place  about  18  months  into  the  24- 
month  contract  period.  These  problems  included  data-system  inadequa¬ 
cies,  failure  to  implement  interventions  for  admissions  and  quality 
objectives  in  a  timely  manner,  and  failure  to  act  on  all  detected  quality 
issues. 

HCFA  officials  have  improved  their  methodology  for  routine  monitoring 
of  PRO  performance  for  the  new  contracts.  However,  this  methodology 
lacks  clear  criteria  defining  what  constitutes  unacceptable  performance 
in  several  areas.  For  example,  although  hcfa  monitoring  personnel  are 
asked  to  judge  if  the  pro  has  satisfactory  working  relationships  with 
providers  and  other  Medicare  contractors,  the  methodology  provides  no 
criteria  defining  what  constitutes  a  satisfactory  relationship. 


HCFA  Monitoring 
Inadequate 


Executive  Summary 


PRO  Program  Funded  at  The  pro  program  is  funded  directly  from  the  Medicare  Trust  Funds,  and 
Legal  Minimum  there  is  a  statutorily  set  minimum  level  of  funding,  hcfa  adopted  this 

minimum  funding  level  as  the  basis  for  its  design  of  the  scope  of  the 
program  for  the  1986-88  contract  period,  gao  found  that  hcfa’s  method¬ 
ology  for  estimating  the  cost  of  individual  pro  contracts  contained  so 
many  uncertainties  that  gao  could  not  determine  whether  these  esti¬ 
mates  were  reasonable  estimates  of  the  costs  to  the  pros  of  carrying  out 
the  contracts.  However,  the  fact  that  virtually  all  contract  awards  dif¬ 
fered  from  estimated  costs  suggests  that  the  contract  negotiation  pro¬ 
cess  may  have  compensated  in  part  for  these  uncertainties. 


Recommendations 


GAO  recommends  that  the  Secretary  of  hhs  direct  the  Administrator  of 
HCFA  to 

assure  that  in  future  pro  evaluations,  the  evaluation  process  has  suffi¬ 
cient  internal  controls  to  assure  that  evaluations  are  consistently 
applied  and  that  decisions  resulting  from  the  evaluations  are  adequately 
documented; 

provide  criteria  to  enable  hcfa  personnel  to  differentiate  between 
acceptable  and  unacceptable  performance  in  routine  monitoring  of  pro 
activities; 

determine  the  scope  of  review  needed  to  adequately  meet  the  program’s 
intent  and  use  this  as  the  starting  point  for  determining  the  program’s 
funding  level;  and 

collect  and  use  adequate  cost  and  performance  data  to  set  each  pro’s 
contract  funding  at  a  level  sufficient  to  provide  the  coverage  determined 
to  be  necessary. 


Agency  Comments 


In  commenting  on  a  draft  of  this  report,  hhs  and  the  American  Medical 
Peer  Review  Association  generally  agreed  with  gao’s  recommendations. 
HHS  said  it  had  taken  and  will  continue  to  take  actions  to  improve  the 
PRO  evaluation  and  funding  processes. 

gao’s  discussions  of  these  comments  are  included  in  the  relevant  chap¬ 
ters,  and  copies  of  the  comments  are  included  as  appendixes  III  and  IV. 
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Chapter  1 _ 

Introduction 


On  June  4, 1986,  the  Chairman  and  Ranking  Minority  Member,  Subcom¬ 
mittee  on  Health,  Senate  Committee  on  Finance,  concerned  that  the  high 
contract  nonrenewal  rate  indicated  problems  either  with  some  Peer 
Review  Organizations  (pros)  or  with  the  Health  Care  Financing  Adminis¬ 
tration’s  (hcfa’s)  program  management,  requested  that  we  review  hcfa’s 
evaluations  of  pros’  performance.  The  evaluations  were  made  to  decide 
whether  to  renew  pro  contracts  without  competition  or  to  require  com¬ 
petition.  The  Chairman  and  Ranking  Minority  Member  were  concerned 
that  the  review  of  pro  performance  be  based  on  objective,  fair,  and  veri¬ 
fiable  measures  of  success  in  meeting  the  program’s  mission.  In  later  dis¬ 
cussions,  the  Subcommittee  also  asked  us  to  determine  how  hcfa  decided 
the  level  at  which  it  would  fund  the  new  pro  contracts. 


Medicare  and  Its 
Utilization  and  Quality 
Review  Policies 


Medicare,  administered  by  hcfa  within  the  Department  of  Health  and 
Human  Services  (hhs),  is  a  health  insurance  program  that  covers  almost 
all  Americans  age  66  and  over  and  certain  individuals  under  65  who  are 
disabled  or  have  chronic  kidney  disease.  The  program,  authorized  under 
title  XVIII  of  the  Social  Security  Act,  provides  protection  under  two 
parts.  Part  A,  the  hospital  insurance  program,  covers  services  of  institu¬ 
tional  providers  of  health  care,  primarily  hospitals.  Part  B,  the  supple¬ 
mentary  medical  insurance  program,  covers  many  noninstitutional 
health  services,  with  most  payments  for  physician  services.  In  1986, 
Medicare  paid  out  $48.8  billion  under  part  A  and  $25.3  billion  under 
part  B  for  health  care  services  and  had  about  31  million  beneficiaries 
enrolled. 


To  assure  that  Medicare  beneficiaries  receive  only  medically  necessary 
and  appropriate  inpatient  hospital  services  of  high  quality,  the  Con¬ 
gress,  as  part  of  the  Social  Security  Amendments  of  1972,  established 
the  Professional  Standards  Review  Organization  (psro)  program.  To 
improve  the  effectiveness  of  that  program,  the  Congress,  in  the  Tax 
Equity  and  Fiscal  Responsibility  Act  of  1982,  redirected  it  and  changed 
its  designation  to  the  Utilization  and  Quality  Control  Peer  Review 
Organization  program,  pros  took  over  responsibility  for  reviewing  the 
necessity,  appropriateness,  and  quality  of  hospital  services  provided 
Medicare  beneficiaries  from  their  predecessor  psros  in  1984. 

Unlike  the  psros,  which  were  mostly  funded  by  annual  grants,  the  act 
required  that  pros  be  administered  and  funded  under  2-year  contracts. 
These  contracts  were  required  to  be  renewable  for  additional  2-year 
terms  at  hhs’s  option.  The  act  specifically  required  that  the  contracts 
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contain  “negotiated  objectives  against  which  the  organization’s  perform¬ 
ance  will  be  judged”  and  also  explicitly  gave  hhs  the  authority  to  evalu¬ 
ate  the  effectiveness  of  the  pros  in  carrying  out  their  contracts,  hcfa 
administers  the  pro  program. 

The  Tax  Elquity  and  Fiscal  Responsibility  Act  of  1982,  which  established 
the  PRO  program,  provided  that  it  be  funded  directly  from  the  Medicare 
Trust  Funds.  This  act  did  not  specifically  address  funding  levels  for 
PROS.  The  Social  Security  Amendments  of  1983  set  a  minimum  funding 
level  based  on  the  costs  of  the  psro  program  in  fiscal  year  1982,  acyusted 
for  inflation.  This  minimum  funding  level  was  modified  by  the  Consoli¬ 
dated  Omnibus  Budget  Reconciliation  Act  of  1985  to  set  minimum  fund¬ 
ing  at  fiscal  year  1986  program  costs,  acyusted  for  inflation. 

In  the  Social  Security  Amendments  of  1983,  the  Congress  modified  the 
way  the  Medicare  Program  pays  most  hospitals  for  inpatient  hospital 
care  by  creating  the  prospective  payment  system  (pps).  As  part  of  this 
law,  it  also  required  that  each  hospital,  as  a  condition  of  payment  by  the 
Medicare  program,  have  an  agreement  with  the  pro  covering  its  area  to 
review  the  quality,  necessity,  and  appropriateness  of  care  provided  to 
Medicare  beneficiaries.  This  law  also  specifically  required  that  the  pro 
review 

•  The  validity  of  diagnostic  information  provided  by  the  hospital,  which 
forms  the  basis  for  most  payments  under  the  new  system; 

•  The  completeness,  adequacy,  and  quality  of  care  provided; 

•  The  appropriateness  of  admissions  and  discharges;  and 

•  The  appropriateness  of  care  provided  to  beneficiaries  for  whom  pay¬ 
ments  are  sought  under  the  outlier  provisions  of  the  act. 

The  PRO  contracts  included  provisions  related  to  the  quality  and  cost 
control  provisions  of  the  pro  and  pps  acts. 

A  PRO  contract  is  in  effect  for  the  Washington,  D.C.,  metropolitan  area, 
Puerto  Rico,  the  Virgin  Islands,  Pacific  island  territories  and  every  state. 


The  PRO  Evaluations 


To  facilitate  monitoring  of  pros’  success  in  meeting  the  requirements  of 
their  contracts,  hcfa  created  the  pro  Monitoring  Protocol  and  Tracking 
System,  or  prompts,  prompts  consisted  of  a  series  of  questions  in  1 3 
areas  of  pro  operations  that  were  to  be  answered  by  hcfa  monitoring 
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personnel  during  quarterly  site  visits.*  In  addition,  medical  review  teams 
re-reviewed  medical  records  as  a  check  on  the  pro’s  ability  to  apply 
review  criteria  and  identify  quality  and  utilization  problems.  The  entire 
PROMPTS  was  to  be  completed  on  the  first  visit,  but  only  those  sections 
that  needed  to  be  updated  or  changed,  such  as  those  on  medical  review 
and  objectives,  were  to  be  completed  on  subsequent  visits.  In  addition, 
areas  found  unsatisfactory  were  to  be  reviewed  during  subsequent 
visits. 

HCFA  also  contracted  with  Systemetrics,  a  research  consulting  firm  spe¬ 
cializing  in  health  care  data  analysis,  to  act  as  the  so-called  SuperPRO. 
Under  this  contract  Systemetrics  re-reviewed  for  each  pro  a  sample  of 
cases  to  evaluate  how  well  the  pro  was  carrying  out  its  contractual 
responsibilities  to  conduct  medical  reviews. 

In  addition,  hcfa  decided  to  conduct  a  special  evaluation  of  pros’  per¬ 
formance  for  the  purpose  of  deciding  whether  to  exercise  the  option  to 
renew  the  contracts  without  competition.  This  evaluation  process  was 
designed  to  use  the  results  of  prompts  and  the  Systemetrics  review,  as 
well  as  information  supplied  directly  by  the  pros.  This  report  is  primar¬ 
ily  concerned  with  hcfa’s  special  evaluation. 


The  Evaluation 
Process 


The  principal  document  used  in  the  evaluation  process  was  the  pro  Eval¬ 
uation  Protocol.  It  consisted  of  three  mjyor  segments: 

The  PRO  self-evaluation  report,  prepared  by  the  pro  together  with 
instructions  for  its  verification  by  hcfa  regional  office  personnel. 

An  independent  analysis,  prepared  by  hcfa  regional  office  personnel, 
summarizing  significant  issues  arising  from  regional  office  monitoring  of 
the  PRO. 

The  evaluation  methodology,  consisting  of  a  single  page  listing  the  16 
elements  of  the  evaluation  along  with  the  maximum  point  value  for  each 
element  (see  p.  13). 


'The  area.**  wen*  implementation,  management,  n'cnasideratioas,  objeetives.  eonfidentiality  tmd  di.s- 
dosure,  medieal  review,  sanetions.  deniai.s,  .s|)et  ialty  hospital  rev  iew,  fraud  and  abuse,  data,  tbe  time¬ 
liness  and  acceptability  of  reports  I’ROs  were  required  to  submit,  and  waiver  of  liability  of  reviews. 
The  last  area  relates  to  PRO  determinatuins  of  whether  hospitals  should  have  knowm  that  the  care 
the  PRO  denied  was  not  txivert'd  by  Medicare  If  the  hositital  could  not  reasonably  have  been  expected 
to  know  the  care  was  not  covered,  it  can  nseive  payment  for  it  tinder  .Medicare's  waiver  of  liability 
program. 


I 

I 
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The  first  step  in  the  pro  evaluation  process  was  preparation  of  the  pro 
self-evaluation  report.-  The  pro  was  asked  to  complete  31  worksheets, 

30  of  which  covered  the  specific  activities  pros  were  required  by  their 
contracts  to  perform.  These  30  worksheets  covered  such  things  as  suc¬ 
cess  in  meeting  targets  for  admission  and  quality  objectives,  perform¬ 
ance  of  admission  and  preadmission  reviews,  production  and  use  of 
profiles,  identification  and  correction  of  utilization  and  quality-of-care 
issues  not  covered  by  objectives,  and  the  pro’s  internal  control  process. 
The  remaining  worksheet  was  designed  to  permit  the  pro  to  cite  any 
achievements  not  specifically  required  by  its  contract.  (See  app.  I  for  a 
complete  list  of  activities  covered  by  the  worksheets.)  The  worksheets 
required  the  pros  to  provide  detailed  data  from  the  first  15  to  17  months 
of  their  contracts  substantiating  their  achievements. 


The  second  step  in  the  process  was  a  review  of  the  pro’s  self-evaluation 
by  the  hcfa  regional  office  staff  responsible  for  monitoring  the  pro  con¬ 
tract.  Regional  office  staff  validated  the  pros’  data  using  the  pros’  files 
and  the  routine  reports  they  had  provided  to  hcfa.  Regional  staff  also 
did  some  analysis  of  the  data  (see  pp.  18-19).  The  pros  were  given  the 
opportunity  to  rebut  the  findings  of  the  regional  staff. 


Regional  office  staff  also  completed  the  independent  analysis  section  of 
hcfa’s  evaluation  instrument.  This  section  was  designed  to  permit  the 
regional  office  staff  to  give  the  evaluation  panels  the  benefit  of  their 
detailed  experience  with  each  pro. 

The  third  step  in  the  contract  renewal  evaluation  process  involved 
designating  an  evaluation  panel  for  each  pro,  which  normally  consisted 
of  five  persons  drawn  from  hcfa’s  central  office,  the  regional  office 
responsible  for  the  pro,  and  one  other  regional  office.  Each  panel  mem¬ 
ber  was  given  a  copy  of  the  completed  evaluation  package,  as  well  as  the 
most  recently  completed  prompts  evaluation,  the  latest  report  from  the 
SuperPRO,  and  a  set  of  scoring  instructions.  Each  panel  member  was  to 
read  over  these  materials  and  independently  evaluate  the  pro  on  the 
basis  of  the  scoring  instructions.  The  panel  then  met  to  prepare  a  con¬ 
sensus  score  and  a  recommendation  for  either  noncompetitive  renewal 
of  the  pro’s  contract  or  competition  for  a  new  contract. 


The  panel  discussed  its  recommendation  with  an  official  in  the  hcfa  cen¬ 
tral  office,  usually  by  telephone.  During  this  discussion,  the  official  was 


-Although  the  self^-valuation  was  optional,  all  but  one  of  the  I’HOs  prepared  one. 
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to  assure  that  the  panel’s  recommendation  was  supported  by  the  evi¬ 
dence.  If  not  satisfied,  the  official  would  ask  the  panel  to  reconsider.  If 
the  panel  scored  the  pro  at  the  “Minimally  Met”  level  in  any  of  the  three 
sections,  the  instructions  specified  that  the  central  office  would  have  the 
final  decision  about  whether  to  have  competition  for  the  contract.  An 
unsatisfactory  score  in  any  of  the  three  sections  meant  that  the  panel 
should  recommend  competition. 

Finally,  the  documentation  of  the  panel’s  deliberations  was  sent  to 
hcfa’s  central  office.  If  central  office  officials  had  questions  or  concerns 
about  a  panel’s  recommendation,  they  would  review  the  panel  documen¬ 
tation.  Program  officials  then  prepared  letters  to  the  evaluated  pro 
informing  it  of  the  decision,  pros  were  permitted  to  appeal  an  unfavora¬ 
ble  decision  and  offer  additional  supporting  data.  All  but  3  of  the  26 
PROS  that  received  unfavorable  decisions  appealed.  However,  only  one 
appeal,  that  of  the  Arizona  pro,  was  successful  in  changing  a  decision. 

Of  the  60  PROS  evaluated,®  26  failed  the  evaluation  and  their  contracts 
were  opened  for  competition,  while  24  were  offered  noncompetitive 
renewal.  In  two  cases,  the  final  decision  on  competition  differed  from 
the  recommendation  of  the  evaluation  panel  (see  app.  II),  and  in  another 
case  it  was  not  clear  what  the  panel  had  recommended  (see  pp.  21-22). 


For  purposes  of  scoring  the  evaluation,  the  16  evaluation  elements  were 
divided  into  three  sections  (see  table  1.1).  The  instructions  to  the  evalu¬ 
ation  panels  required  that  the  pro  achieve  a  satisfactory  score  in  all 
three  sections  for  a  recommendation  for  noncompetitive  contract 
renewal. 


®Four  PRO  ('(intracts  were  terminated  during  the  1984-86  contract  period  and  were,  'herefore,  not 
included  in  the  evaluation. 
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Tabi*  1.1:  PRO  Evaluation  Scoring 


Objectives,  Scope,  and 
Methodology 


Element  name 

Score 

Maximum 

Minimum 

satisfactory 

Section  1 — Meeting  Objectives 

Admission  objectives 

200 

130 

Quality  objectives 

200 

130 

Admission  objectives  interventions 

50 

33 

Quality  objectives  interventions 

50 

33 

Total 

500 

326 

Section  2 — Required  Review  Activities 

Revie\w  types  required  by  contract 

100 

65 

Profiling 

25 

17 

Waiver  activity 

25 

17 

Intensified  review 

25 

17 

Total 

175 

116 

Section  3 — PRO  Management 

Abuse  referrals 

25 

17 

Sanctions 

75 

49 

Reconsiderations 

25 

17 

Utilization  problems 

50 

33 

Quality-of-care  issues 

50 

33 

Private  review 

25 

17 

Internal  controls 

25 

17 

Impact  outside  objectives 

50 

50 

Total 

325 

224 

Because  the  number  of  points  in  each  section  were  not  equal  and 
because  the  pro  had  to  achieve  a  score  of  satisfactory  for  each  section  to 
receive  a  recommendation  of  noncompetitive  contract  renewal,  the 
number  of  points  in  a  given  element  did  not  represent  its  relative  impor¬ 
tance  to  the  total  evaluation. 


As  requested  by  the  Chairman  and  Ranking  Minority  Member  of  the 
Subconunittee  on  Health,  Senate  Committee  on  Finance,  our  review 
objectives  were  to  determine  (1)  how  hcfa  monitored  pros’  performance 
in  meeting  the  requirements  of  their  contracts,  (2)  the  methodology  hcfa 
used  in  deciding  whether  to  renew  pro  contracts  without  competition, 
and  (3)  how  hcfa  decided  on  the  funding  level  of  the  new  pro  contracts. 
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We  were  also  requested  to  determine  how  the  factors  used  in  the 
renewal  evaluation  process  compared  to  the  pros’  contractual  require¬ 
ments.  We  found  that  with  one  exception,  all  elements  in  the  evaluation 
could  be  traced  to  a  contractual  requirement.  The  exception  was  the  ele-  i 
ment  on  impact  outside  objectives,  hcfa  officials  told  us  that  this  ele¬ 
ment  had  been  included  to  give  the  pros  a  way  to  gain  credit  for 
accomplishments  in  reducing  utilization  or  improving  quality  of  care 
that  were  not  specifically  covered  by  contractual  requirements.  Because 
we  found  no  problems,  we  df)  not  discuss  this  area  in  the  remainder  of 
the  report. 

Finally,  we  were  asked  whether  hcfa  had  clearly  indicated  to  the  PRtis 
the  type  of  documentation  required  by  the  evaluation  process.  On  Octo¬ 
ber  9, 1985,  about  5  weeks  before  the  evaluation  began  for  the  first 
group  of  PROS,  HCFA  gave  the  draft  evaluation  protocol  to  the  pros  and  to  | 
the  American  Medical  Peer  Review  Association  (ampra),  the  pro  trade 
organization,  for  conunent.  This  action  gave  the  pros  advance  notice  of 
the  documentation  requirements  of  the  evaluation.  Those  pros  with 
whom  we  discussed  the  question  thought  that  they  had  received  ade¬ 
quate  advance  notice  of  the  evaluation. 

1 

As  part  of  our  assessment  of  the  methodology  hcfa  used  to  decide 
whether  to  renew  pro  contracts  without  competition,  we  also  considered 
hcfa’s  internal  controls  over  the  evaluation  development  and  implemen¬ 
tation.  Internal  controls  are  the  combination  of  policies  and  procedures 
used  by  managers  to  help  assure  that  their  programs  are  effectively  and 
consistently  managed.  We  assessed  whether  hcfa’s  internal  controls  pro¬ 
vided  reasonable  assurance  that  the  renewal  evaluation  methodology 
was  consistent  with  the  pro  contracts  and  fairly  and  consistently  j 

applied  to  evaluate  pros. 

To  examine  the  renewal  evaluation  process  in  detail,  we  Judgmentally 
selected  6  of  the  41  pros  whose  evaluations  had  been  completed  as  of 
June  12,  1986.  They  were  selected  to  cover  examples  of  evaluations  i 

resulting  in  satisfactory  scores  (2)  and  unsatisfactory  scores  (4).  In  mak¬ 
ing  the  selections  we  considered  information  supplied  by  ampra  about 
evaluations  it  believed  might  have  had  problems.  We  selected 

•  Medical  Review  of  North  Carolina,  Inc.  (North  Carolina): 

•  Mississippi  Foundation  for  Medical  Care,  Inc.  (Mis.sissippi); 

•  Kentucky  Peer  Review  Organization  (Kentucky); 

•  Peer  Review  Systems.  Inc.  (Ohio);  j 

•  Utah  Professional  Standards  Review  Organization  ( I  'tah );  and 
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•  Indiana  Peer  Review  Organization  ( Indiana ).  | 

( 

During  our  work  we  noted  that  the  evaluation  panel  had  not  recom¬ 
mended  nonc’ompetitive  renewal  of  the  Professional  Foundation  for 
Health  Care  (Florida),  but  this  pro  was  noncompetitively  renewed. 

Therefore,  we  reviewed  those  aspects  of  the  evaluation  of  this  pro 
related  to  the  decision  for  noncompetitive  renewal. 

In  addition,  we  analyzed  the  scoring  of  46  evaluations  for  inconsisten¬ 
cies  with  the  instructions,^  and  observ-ed  the  contract  negotiation 
between  hcfa  and  the  California  pro. 

We  interviewed  hcpa  and  pro  officials,  as  well  as  other  interested  par¬ 
ties;  examined  documents;  analyzed  cost  information  supplied  by  hoka; 
and  compared  relevant  laws,  regulations,  and  manual  instructions  with 
the  evaluation  instruments.  We  also  compared  the  methodology  of 
hcfa’s  routine  monitoring  of  pros  with  the  methodology  for  renewal 
evaluation. 

Our  fieldwork  was  done  from  June  1986  through  March  1987  in  accor¬ 
dance  with  generally  accepted  government  auditing  standards. 
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Our  review  of  the  development  and  implementation  of  the  pro  renewal 
evaluation  process  showed  that  instructions  were  inconsistent,  incor¬ 
rect,  or  not  properly  implemented  by  the  evaluation  panels  and  that  doc¬ 
umentation  of  the  evaluation  results  was  not  adequate.  These  problems 
resulted  in  incorrect  scores  for  individual  elements  of  the  evaluation  and 
an  undocumented  decision  to  offer  noncompetitive  contract  renewal  to  a 
PRO  that  had  not  been  recommended  for  renewal  by  its  evaluation  panel. 
The  documentation  available  to  us  for  the  seven  evaluations  we 
examined  was  insufficient  to  enable  us  to  determine  that  these  problems 
led  to  a  clearly  inappropriate  renewal  or  nonrenewal  decision. 

HCFA  PRO  program  managers  generally  acknowledged  that  the  problems 
we  identified  existed,  and  attributed  them  to  inadequate  time  and  staff 
to  implement  and  manage  the  program,  hcfa  will  need  to  make  contract 
renewal  decisions  every  2  years,  and  we  believe  that  hcfa  should  assure 
that  adequate  internal  controls  are  established  so  that  in  future  evalua¬ 
tions  of  PRO  jierformance,  (1)  the  process  is  internally  consistent,  (2)  the 
evaluation  instructions  are  clear  and  consistently  applied  to  all  pros,  and 
(3)  evaluation  results  are  adequately  documented. 


HCFA  Lacks 
Documented  Internal 
Controls  Over  the 
Evaluation  Process 


HCFA  did  not  have  an  internal  control  system  to  assure  the  systematic 
documentation  of  all  phases  of  the  development  and  implementation  of 
the  process  used  to  evaluate  contractor  performance  to  decide  whether 
to  renew  a  contract  without  competition.  Such  documentation  is  neces¬ 
sary  to  support  the  organization’s  position  in  event  of  a  challenge  to  an 
individual  evaluation  or  to  the  evaluation  process  as  a  whole. 


A  HCFA  official  told  us  that  the  evaluation  document  and  the  panel 
instructions  were  developed  by  a  task  force  that  included  the  two  offi¬ 
cials  most  directly  responsible  for  the  implementation  of  the  evaluation 
process.  This  task  force  appears  to  have  been  the  principal  internal  con¬ 
trol  for  assuring  that  all  portions  of  the  evaluation  document  and 
instructions  were  consistent  with  one  another.  However,  according  to 
this  official,  the  task  forc*e  did  not  document  its  decisions. 

The  main  internal  control  'rocess  for  assuring  that  the  evaluation 
panels  implemented  the  evaluation  consistently  and  had  sufficient  evi¬ 
dence  to  support  their  recommendations  centered  on  a  requirement  that 
the  panels  di.scu.ss,  usually  by  telephone,  their  finding  with  one  of  the 
two  central  office  officials  responsible  for  implementing  the  evaluation, 
who  also  participated  in  the  evaluation  development  task  force.  How¬ 
ever.  this  pr(K'ess  was  not  documented.  Except  for  the  requirement  that 
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letters  to  the  pros  informing  them  of  the  evaluation  outcome  be  initialed 
by  a  responsible  official,  hcfa  officials  had  no  documented  proc-ess  for 
assuring  that  the  panels  had  complied  with  their  instructions  and  for 
assuring  that  the  evaluation  decision  was  approved  by  responsible  offi¬ 
cials  after  the  panels  sent  their  ret'ommendations  and  supporting  d(K-u- 
mentation  to  hcfa’s  central  office. 


There  were  inconsistencies  between  the  instructions  to  the  regional 
offices  for  verifying  the  quality  objectives  information  supplied  in  the 
Evaluation  Protocol  and  the  instructions  to  the  panel  for  scoring  the 
quality  objectives  element.  The  verification  instructions  gave  a  less 
stringent  standard  for  what  constituted  acceptable  performance  in  this 
area,  while  the  instructions  to  the  evaluation  panel  did  not  address  a 
contractually  required  severity  index  intended  to  weight  the  objectives 
for  the  severity  of  the  problem  addressed. 

The  instructions  to  hcfa  regional  office  personnel  for  verifying  and  ana¬ 
lyzing  the  quality  objectives  information  supplied  by  the  pros  in  the  self- 
evaluation  stated  that  these  elements  were  to  be  considered  satisfactory 
if  the  difference  between  total  expected  and  actual  quality  impact  was 
not  more  than  5  percent  and  four  of  five  objectives  were  found 
satisfactory. 

This  instruction  required  hcfa  regional  office  monitoring  personnel  to 
calculate  a  severity-weighted  measure  of  the  total  impact  of  the  quality 
improvements  effected  by  the  pro  through  its  contracted  quality  objec¬ 
tives.  The  PRO  contract  contained  severity  index  numbers  that  were 
assigned  to  each  quality  objective  to  indicate  the  severity  of  the  pn>blem 
addressed  by  the  quality  objective.  A  severity-weighted  difference 
between  the  actual  and  expected  impacts  was  computt*d  for  each  quality 
objective,  as  well  as  for  the  total  of  all  quality  objcetives  By  defining 
acceptable  performance  in  this  manner,  a  pro  could  have  performt*d 
acceptably  in  the  overall  element  even  if  it  had  faiU*d  to  achieve  its  tar¬ 
get  for  one  quality  objective. 

However,  the  standards  for  acceptable  performanct*  on  quality  objts - 
tives  in  the  instructions  given  to  the  panels  were  somewhat  more  strin¬ 
gent.  These  instnictions  define  minimally  acceptable  |M‘rformance  as  the 
PRO  meeting  all  omtracted  targets,  unlc'ss  failure  to  mcs't  them  was  "not 
because  of  the  lack  of  action  by  the  pro.  "  While  the  instruction  to  the 
panel  was  stricter  than  that  to  hcfa  regional  office  |H‘rsonnel,  it  did  n(»t 


17 


OAO  HRI>aSI,iPROKvaluaU<iiw 


CkapterZ 

Inaafllciait  laternal  Coatrala  Ovrr  the 
iMMwal  EvalnaUoa  Proee— 


use  the  severity  index  in  evaluating  the  pro’s  performanc'e  despite  its 
inclusion  in  the  contract. 

Incx>n8istent  instructions  caused  ik’KA  regional  offite  monitoring  person¬ 
nel  and  the  evaluation  panel  to  ctime  to  different  conclusions  regarding 
the  acceptability  of  the  Utah  pro’s  performance  on  its  quality  objectives 
The  I’tah  pro  failed  to  achieve  one  of  its  quality  objectives  by  a  large 
margin.  However,  it  achieved  its  other  quality  objectives  by  large  mar¬ 
gins.  Its  overall  performance,  when  calculated  by  the  methodology  given 
the  HC'FA  regional  office  personnel,  was  c-onsiderably  bt*tter  than  that 
required  to  have  performcKl  satisfactorily  in  this  element  under  the  cri¬ 
teria  in  the  verification  instrurtions  uck.a  monitoring  personnel  there¬ 
fore  rated  the  i*Ko  satisfactory  in  the  quality  objective's  area.  However, 
the  evaluation  panel,  as  called  for  in  its  instructions,  rated  the  i*ro 
unsatisfactory  in  the  evaluation’s  quality  objectives  element  because  it 
had  failed  to  achieve  one  objective 

When  we  discussed  thjs  inc-onsistency  of  instructioas  with  program  offi¬ 
cials.  they  agreed  that  it  existed,  and  noted  that  they  had  not  had  the 
time  or  staff  resourct*s  to  ctxirdinate  all  phases  of  the  development  of 
the  evaluaticHi  prexess  as  they  wcHild  have  wished  to 


Inconsistency  in 
Instructions  on 
Profiling 


There  was  als«>  an  incxm.sistency  between  the  last  nut  ions  to  the  panel 
and  the*  verification  lastnutioas  to  regional  office  (X'rsonnel  for  the  pro¬ 
filing  element  The*  pro  contract  scoja*  of  work  required  that  the  pro 
have  the'  capability  of  developing  profile's  for  patients,  physicians,  and 
otlier  pnvide'rs  within  days  after  the  contract  we*nt  into  effes  t  The 
instnictions  to  the*  e'valuation  panel  f(»r  this  elemient  slate  that  to  rcneive 
any  ciedit  for  this  c'le'iue'iit,  the  pro  must  have*  at  a  minimum  fulfille'd  the 
resjuirements  of  the  contrad  aftcT  an-eplancc*  of  a  corns  tive  action  plan 
by  the'  ri'gional  offies* 

However,  the  instnictions  to  re>gional  office*  |ie*rsonne‘l  for  ve'rifiialHin  of 
the  data  suppliesl  by  the*  pro  m  the*  pro  re*j»ort  state*  that  the*  pro  is  to  be* 
found  de'ficie'Ht  if 

It  was  unable*  te»  rese'ive*  and  prisess  data  from  the*  mte*nne*diary  within 
4ri  days  of  the  e'ffes  tive*  date*  eif  its  e  eintrae  t  e>i 

the*  first  pnifile  nin  was  ne»t  proce-ssesl  fe»r  use*  t»\  the*  pro  within  ti 
meenths  freim  the*  e*ffes  tive*  elate*  of  the*  (  eintrae  t 
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This  seems  to  be  more  stringent  in  one  respect  and  less  stringent  in 
another  than  contract  requirements.  As  noted  above,  the  contract 
requires  that  the  pro  have  the  capability  to  run  profiles  within  45  days 
of  the  effective  date  of  the  contract,  which  seems  stricter  than  the 
standard  to  be  able  to  receive  and  process  data  within  45  days  in  the 
verification  instructions  to  regional  office  personnel.  On  the  other  hand, 
the  contracts  do  not  mention  requiring  the  pros  to  run  a  profile  for  use 
within  6  months  or  any  other  time  frame. 

Although  we  did  not  identify  any  instance  where  this  inconsistency 
affected  the  outcome  of  an  evaluation,  such  an  inconsistency  raises  the 
possibility  of  inconsistent  evaluations  of  pros  since  three  out  of  five 
members  of  the  evaluation  panels  were  usually  regional  office  monitor¬ 
ing  personnel. 


There  was  an  error  in  the  instructions  to  the  evaluation  panel  for  the 
sanctions  element.  Although  hcfa  officials  were  aware  of  this  error,  the 
panel  instructions  were  not  revised.  Instead,  h('KA  program  officials 
attempted  to  insure  that  panels  did  not  follow  the  erroneous  instruction 
through  their  procedures  for  reviewing  the  panel’s  decision.  However,  in 
one  case,  a  PRO  failed  the  pro  management  section  of  the  evaluation 
because  the  panel  used  the  incorrect  instructions,  and  based  on  this, 
competitive  contract  renewal  was  recommended.  The  error  was  not  cxir- 
rected  until  the  pro  appealed  the  decision  to  compete. 

The  1984  contract  required  that  pros  initiate  sanction  pn)ceedings 
against  health  care  practitioners  whom  they  found  rendering  services 
that  do  not  meet  professionally  recognized  standards  of  health  care. 
ik'Ka's  iastructions  to  the  evaluation  panels  requiri*d  that  for  the  pro  to 
be  si-ored  as  "fully  met"  in  this  element,  it  had  to  have  a  sanction  case  in 
prtKess.  In  order  to  receive  the  full  75  points  for  this  element,  defimnl  as 
"exceeded  met,"  the  PW*  was  required  to  have  submittcKl  a  sanction  case* 
to  iins's  Office  of  the  Inspec  tor  General  for  adjudication. 

IICM  officials  told  us  that  in  light  of  the  (xissibility  that  a  I'Ro  might  not 
have  ftmnd  any  sanctionabU*  cases,  it  was  unreasonable*  to  hold  i*Kos  to 
this  standard  They  said  that  they  had  discov<*rt‘d  this  problem  with  the 
lastnulions  to  the  |>anel  whc*n  the*  first  group  of  pros  was  Iwing  evalu 
ated  iic  KA  dcs  ided  that  pros  had  to  lx*  giv»*n  a  fully  met  score  lor  sane 
tions  if  no  sanctionable  problems  had  lHs*n  found  by  the  pro.  the 
Supc*rPRo,  or  IR  KA  regional  offus*  |x*rsonnel.  or  if  a  problem  had  lxs*n 
identificHl  and  the  pro  had  taken  some  action  toward  attempting  to 


Pmv  !• 


OAO  HUVNN  l:l  PKO  KvaluaOfNW 


Chi4l>ter2 

Inaufllcient  Internal  Controla  Over  the 
Benewal  Evaluation  Proceea 


I 


Incorrect  Scoring  of 
PROS  by  Panels 


resolve  it.  Although  the  written  instructions  were  not  modified,  hcfa 
officials  said  that  they  had  attempted  to  assure  that  the  stricter  stand¬ 
ard  was  not  enforced  when  the  panels  discussed  their  recommendations 
with  one  of  two  central  office  officials.  Where  the  panels  had  applied 
the  more  stringent  standard  in  reviewing  the  first  group,  central  office 
officials  told  us  that  they  had  attempted  to  assure  that  this  was  not  the 
sole  reason  for  the  pro’s  failing  the  evaluation. 

However,  in  the  case  of  the  Utah  pro,  which  was  among  the  first  group 
of  PROS  evaluated,  the  panel  gave  the  pro  a  zero  score  in  this  element 
even  though  no  sanctionable  problems  had  been  identified.  Had  the 
panel  given  the  pro  a  minimally  passing  score  for  this  element,  it  would 
have  achieved  a  passing  score  on  all  three  sections  of  the  evaluation. 
Nevertheless,  hcfa  central  office  officials  accepted  the  panel's  recom¬ 
mendation  to  require  competition  for  the  contract.  However,  when  the 
PRO  appealed  the  decision,  hcfa  central  office  officials  accepted  the  pro’s 
rebuttal  of  this  point.  Despite  this  reversal  on  the  sanctions  issue,  hcfa 
did  not  overturn  the  panel’s  recommendation  because  new  information 
submitted  by  the  pro  showed  that  it  had  found  several  quality -of -care 
problems  that  it  had  not  intervened  to  correct. 


In  addition,  in  some  instances,  panels  gave  pros  individual  element 
scores  that  were  inconsistent  with  the  instructions.  In  13  of  16  evalua¬ 
tion  elements,'  the  instructions  to  the  panel  indicated  that  specific  scores 
were  to  be  given  for  each  of  three  defined  levels  of  performance.  For 
example,  for  the  profiling  element,  the  score  for  "fully  met”  was  set  as 
26  points,  the  “minimally  met”  score  as  17  points,  and  the  “unsatisfac¬ 
tory”  score  as  0  points.  Thus,  only  these  three  scores  should  have  been 
given  for  this  element. 

However,  in  46  instances  spread  over  1 5  evaluations,  panels  had  given 
element  scores  other  than  the  scores  required  by  the  instructions,  e.g., 

20  points  for  the  profiling  element,  hcfa  program  officials  told  us  that 
such  scores  were  not  permitted  by  the  instructions,  and  a  senior  official 
who  received  many  of  the  telephone  calls  from  the  evaluation  panels 
told  us  that  if  they  had  come  to  his  attention  during  the  evaluation  pne 
cess,  he  would  have  asked  the  panels  to  correct  them. 


'  For  the  iKher  thm*  ekmenl.s  Itw  inMniilkitut  explw  illy  (■■miillixl  »  raiiK''  "f  (nr  a  (h'liiHal 
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We  found  no  instance  where  errors  of  this  type  affected  the  outcome  of 
the  evaluation.  However,  the  fact  that  such  errors  occurred  so  often, 
and  that  they  occurred  within  every  group  of  pros  evaluated,  indicates 
to  us  that  the  instructions  to  the  panel  on  this  point  were  unclear.  It  also 
shows  that  hcfa’s  measures  for  monitoring  the  evaluation  process — 
having  the  panel  officials  telephone  a  hcfa  central  office  official  to  dis¬ 
cuss  decisions  and  supporting  evidence — were  not  adequate  to  prevent 
the  panels  from  assigning  incorrect  scores.  A  stronger  internal  control, 
such  as  a  requirement  that  all  panel  decisions  and  supporting  documen¬ 
tation  be  reviewed  and  approved  in  writing  by  central  office  officials, 
could  have  alerted  them  to  this  problem. 


During  our  review,  we  found  that  the  decisions  to  noncompetitively 
renew  the  Florida  and  Indiana  pros’  contracts  were  inadequately  docu¬ 
mented.  Although  these  do  not  appear  to  have  affected  the  outcome  of 
the  evaluations,  lack  of  documentation  indicates  inadequate  manage¬ 
ment  oversight  and  control  over  the  evaluation  process.  Senior  pro  pro¬ 
gram  staff  acknowledged  these  documentation  problems  and  attributed 
them  to  inadequate  internal  controls  over  the  process. 


The  evaluation  results  for  the  Florida  pro  were  poorly  documented  for 
both  the  panel  and  postpanel  phases  of  the  evaluation.  The  evaluation 
panel  (is  d  not  recommend  noncompetitive  renewal  of  the  Florida  con¬ 
tract.  The  contract;  was  noncompetitively  renewed,  and  we  could  find  no 
evidence  that  responsible  program  officials  had  explicitly  decided  to 
renew  the  Florida  pro  noncompetitively.  However,  we  were  unable  to 
determine  whether  contract  renewal  was  the  appropriate  action. 

Evaluation  panel  documentation  showed  that  the  Florida  pro  had  failed 
section  2  (Required  Review  Activities)  of  the  evaluation.  Discussions 
with  the  chairman  and  senior  central  office  member  of  the  Florida  pro 
evaluation  panel  showed  that  the  panel  was  deeply  divided  over 
whether  the  Florida  pro  should  be  offered  noncompetitive  renewal  or 
the  contract  recompeted.  Initially,  the  panel  had  arrived  at  a  tentative 
recommendation  for  competition.  However,  when  panel  members  called 
a  senior  official  in  the  central  office  to  discuss  the  panel’s  findings,  that 
official  questioned  the  panel's  basis  for  this  recommendation.  According 
to  these  two  members  of  the  panel,  they  had  the  impression  that  they 
were  being  ordered  to  pass  the  Florida  pro.  They  also  said  they  were 
instructed  by  this  official  to  review  the  Florida  pro's  technical  proposal 
for  the  new  contract.  Normally,  review  of  the  technical  proposal  would 
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be  done  only  if  a  recommendation  for  noncompetitive  renewal  had  been 
made. 


Although  it  is  clear  that  the  panel  did  not  recommend  that  the  Florida 
PRO  be  renewed  noncompetitively,  the  available  evidence  is  ambiguous 
on  whether  the  panel  recommended  competition  for  this  contract  or 
whether  it  returned  no  recommendation.  The  Florida  evaluation  panel 
documentation  contained  two  panel  recommendation  sheets.  The  first, 
dated  March  17, 1987,  indicated  that  the  panel’s  decision  was  for  con¬ 
tract  recompetition,  but  it  was  not  signed  by  the  panel  members.  The 
second,  dated  March  18, 1987,  was  signed,  but  no  recommendation  was 
indicated.  The  panel  chairman  told  us  that  he  was  unable  to  recall  if  the 
panel  had  reached  a  final  decision  about  what  recommendation  to  make 
after  the  phone  call  to  the  central  office  official.  However,  the  senior 
central  office  member  of  the  panel  told  us  that  the  panel  had  decided  not 
to  change  its  mind  about  its  recommendation  for  recompetition  of  the 
contract. 


Although  the  panel  had  not  recommended  noncompetitive  renewal,  hcfa 
renewed  the  Florida  pro’s  contract  without  competition.  We  were  unable 
to  find  any  documentation  of  this  decision  except  the  letter  to  the  Flor¬ 
ida  PRO  offering  it  noncompetitive  renewal.  Furthermore,  neither  of  the 
two  HCFA  officials  most  directly  in  charge  of  the  evaluation  could  recall 
having  made  the  decision  to  offer  noncompetitive  renewal  to  the  pro. 
However,  one  of  them  told  us  that  he  believed  that  the  pro  should  have 
passed  the  evaluation  and  been  noncompetitively  renewed.  He  said  that 
many  of  the  pro’s  difficulties  could  be  traced  to  the  fact  that  it  was 
unable  to  obtain  data  from  the  intermediaries  in  Florida  for  several 
months,  a  problem  for  which  hcfa  was  responsible.  In  addition,  he  noted 
that  the  pro  had  performed  review  despite  this  lack  of  data,  which  was 
not  required  by  the  contract.  He  said  that  he  would  have  recommended 
noncompetitive  contract  renewal  for  the  pro  to  his  superiors  had  the 
question  ever  come  up. 


The  only  documentation  we  found  for  the  decision  to  offer  noncompeti¬ 
tive  renewal  to  the  Florida  pro  was  a  draft  of  the  letter  to  the  pro,  ini¬ 
tialed  by  this  HCFA  official.  From  our  discussions  with  him,  it  does  not 
appear  that  he  realized  at  the  time  that  his  approval  of  this  draft  letter 
was  in  effect  a  decision  to  offer  noncompetitive  renewal  to  the  Florida 
PRO  despite  the  fact  that  the  evaluation  panel  had  not  recommended 
this.  However,  he  said  that  he  believed  that  noncompetitive  renewal  was 
the  proper  outcome. 
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The  HCFA  official  who  actually  drafted  the  letter  to  the  pro  offering  it 
noncompetitive  renewal  could  not  remember  who.  if  anyone,  had  told 
her  to  draft  this  letter.  However,  she  said  that  she  had  received  the 
results  of  the  panel's  review  of  the  Florida  pro’s  technical  proposal  for 
the  new  contract,  which  she  normally  got  only  in  cases  when  the  pro 
had  passed  the  evaluation.  She  said  that  she  did  not  always  see  the 
panel’s  renewal  evaluation  documentation.  Although  the  official  could 
not  recall  all  the  details,  it  appears  that  she  assumed  that  the  panel 
review  of  the  technical  proposal  meant  that  the  panel  had  recommended 
noncompetitive  renewal  and  drafted  the  letter  accordingly. 

We  were  unable  to  determine  whether  the  decision  to  renew  the  Florida 
PRO  noncompetitively  was  reasonable.  The  rationale  advanced  by  one 
UCFA  official  that  the  pro  should  have  been  renewed  because  its  prob¬ 
lems  were  due  to  a  lack  of  good  data  from  the  Florida  intermediaries 
seems  reasonable.  On  the  other  hand,  other  pros  that  were  not  renewed 
also  had  problems  obtaining  data  from  intermediaries.  In  any  case,  as 
nearly  as  we  can  determine,  neither  the  panel  nor  central  office  officials 
explicitly  made  a  renewal  decision.  In  effect,  the  decision  was  implicitly 
made  when  a  central  office  official  initialed  the  draft  of  the  letter  to  the 
PRO  offering  it  noncompetitive  renewal  of  its  contract 

The  confusion  surrounding  the  evaluation  for  this  pro  suggests  to  us 
that  management  controls  over  that  part  of  the  evaluation  process 
occurring  after  the  panels  had  returned  their  decisions  were  not  ade¬ 
quate.  Specifically,  hcfa  had  not  established  a  formal  process,  including 
appropriate  documentation,  requiring  key  central  office  officials  to 
review  and  approve  each  evaluation  decision. 


Another  type  of  documentation  problem  oc:curred  in  the  evaluation  pro¬ 
cess  for  the  Indiana  pro.  The  evaluation  panel’s  documentation  indicated 
that  the  pro  had  failed  section  1  of  the  evaluation  by  a  substantial  mar¬ 
gin,  in  part  because  of  a  poor  score  in  the  quality  objectives  element. 

The  pro  passed  sections  2  and  3.  Despite  apparently  failing  section  1, 
which  according  to  the  panel  instructions  should  have  resulted  in  a  rec¬ 
ommendation  for  competition,  the  panel  recommended  that  the  contract 
be  renewed  noncompetitively.  The  reasons  for  this  decision  were  not 
documented,  nor  were  we  able  to  clarify  the  situation  through  discus¬ 
sions  with  panel  members. 

We  did  identify  a  document,  not  part  of  the  official  panel  documenta¬ 
tion,  which  indicated  that  the  score  for  section  1  had  bet'n  raised  to  the 
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passing  level.  The  original  score  for  this  element,  66  points,  had  been 
crossed  out  and  174  points  substituted.  This  change  was  annotated  “per 
conf[erence]  call  3/21.”  A  change  of  this  amount  would  have  increased 
the  score  for  section  1  from  failing  to  minimally  met  and  caused  the  pro 
to  pass  all  three  sections  of  the  evaluation. 

The  HCFA  regional  office  official  who  had  been  responsible  for  monitor¬ 
ing  this  PRO  told  us  that  this  document  was  an  informal  working  docu¬ 
ment  used  by  the  panel  to  record  its  findings  during  its  deliberations.  He 
told  us  that  the  score  had  been  changed  as  a  result  of  a  conference  call 
made  from  the  Chicago  Regional  Office  after  the  panel  had  disbanded. 

He  said  that  the  participants  in  that  call  had  been  he,  his  immediate 
supervisor  in  hcfa’s  Chicago  Regional  Office,  and  the  director.  Division 
of  Program  Operations,  within  hcfa’s  Health  Standards  and  Quality 
Bureau,  the  division  responsible  for  pro  program  management.  How¬ 
ever,  neither  of  the  other  two  officials  could  recall  the  conference  call. 
Furthermore,  one  member  of  the  panel  we  talked  to  had  no  knowledge 
of  this  change,  nor  did  the  other  central  office  official  who  was  responsi¬ 
ble  for  directing  the  evaluation  process.  In  addition,  we  did  not  find  any 
other  documentation  of  a  change  in  score  for  the  Indiana  pro. 

In  spite  of  this,  hcfa  officials  believe  the  change  in  score  was  appropri¬ 
ate  and  said  that  the  panel  documentation  should  have  been  corrected  to 
reflect  the  change.  In  any  case,  it  could  have  had  no  effect  on  the  panel 
recommendation,  because  the  panel  had  disbanded  before  the  conversa¬ 
tion  during  which  the  modification  was  discussed,  and  one  panel  mem¬ 
ber  we  talked  to  was  unaware  of  it.  The  panel  had  already  recommended 
that  the  pro’s  contract  be  renewed  noncompetitively  despite  the  fact 
that  it  had  failed  section  1 .  The  central  office  reviewed  the  recommenda¬ 
tion  and  accepted  it.  This  possible  change  in  score  had  no  ascertainable 
effect  on  the  outcome  of  the  evaluation.  However,  as  in  the  case  of  Flor¬ 
ida,  the  failure  of  program  officials  to  adequately  document  their 
actions  suggests  that  hcfa’s  internal  controls  over  the  evaluation  process 
were  inadequate. 


The  PRO  evaluation  process  was  internally  inconsistent,  had  errors,  and 
was  inconsistently  applied,  and  evaluation  decisions  were  not  ade¬ 
quately  documented.  We  believe  this  occurred  because  of  inadequate 
internal  controls  over  the  evaluation  process,  which  did  not  include  (1)  a 
documented  central  office  review  of  each  panel’s  decision  and  its  sup¬ 
porting  documentation  and  (2)  an  approval  process  involving  key  cen¬ 
tral  office  officials.  Although  we  were  unable  to  document  any  instances 
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where  these  problems  led  to  an  inappropriate  decision  to  offer  or  with¬ 
hold  noncompetitive  contract  renewal,  the  potential  for  such  errors 
existed. 

HCFA  program  officials  generally  agreed  that  the  matters  we  found  were 
problems  and  attributed  them  to  inadequate  resources  to  implement  and 
manage  the  program.  We  believe  that  hcfa  management  should  assure 
that  future  evaluations  of  pros  are  designed  with  sufficient  internal  con¬ 
trol  mechanisms  to  assure  consistent,  adequately  documented 
evaluations. 

Recommendation 

We  recommend  that  the  Secretary  of  hhs  instruct  the  Administrator  of 
hcfa  to  assure  that  in  future  PRO  evaluations,  the  evaluation  process  has 
sufficient  internal  controls  to  assure  that  evaluations  are  consistently 
applied  and  that  decisions  resulting  from  them  are  adequately 
documented. 

In  a  letter  dated  September  4, 1987,  the  hhs  inspector  general  stated  that 
HHS  agreed  with  our  recommendation  and  that  steps  have  been  taken  to 
improve  the  evaluation  internal  controls  based  on  hcka’s  experience  dur¬ 
ing  the  first  contract  evaluation  process  (see  app.  III).  As  an  example  of 
improvement,  hhs  said  that  the  summary  sheets  prepared  by  the  review 
panels  will  be  controlled  to  document  the  nature  of  the  decision-making 
process,  hhs  also  said  that  it  will  continue  to  improve  internal  controls. 

In  an  August  30,  1987,  letter,  the  executive  vice  president  of  ampra 
agreed  with  our  findings  and  recommendation  (see  app.  IV). 


Agency  Comments  and 
Our  Evaluation 
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HCFA  had  not  identified,  and  therefore  had  not  required  pros  to  correct, 
many  perfonnance  problems  before  the  contract  renewal  evaluations. 
These  evaluations  took  place  about  18  months  into  the  24-month  con¬ 
tract  period.  This  failure  to  detect  and  correct  performance  problems 
was  due  in  part  to  inadequacies  in  hcfa’s  measures  for  routine  monitor¬ 
ing  of  PRO  performance,  which  did  not  evaluate  pro  performance  in 
important  areas  covered  by  the  contract  evaluation,  and  may  have 
diminished  the  program’s  ability  to  protect  Medicare  and  its 
beneficiaries. 

HCFA  officials  became  aware  of  the  deficiencies  in  pro  monitoring  when  it 
became  apparent  that  about  half  of  the  pros  were  failing  the  renewal 
evaluations.  They  told  us  that  they  were  attempting  to  improve  the  rou¬ 
tine  monitoring  of  the  pros  by  revising  and  improving  the  main  instru¬ 
ment  for  monitoring  pro  performance — prompts — and  by  increasing  the 
amount  of  data  that  the  pros  were  required  to  report.  The  new 
PROMPTS — called  prompts-2 — was  completed  on  February  27,  1987, 
about  7  months  into  the  new  contract  cycle  for  some  pros,  prompts-2  is 
designed  to  monitor  pro  performance  in  the  important  areas  covered  by 
the  contract  evaluation.  However,  in  some  cases  it  lacks  criteria  to  help 
HCFA  regional  office  monitoring  personnel  differentiate  between  accepta¬ 
ble  and  unacceptable  performance. 


All  but  one  of  the  six  pros  whose  evaluations  we  examined  had  perform¬ 
ance  problems  that  were  not  detected  by  hcfa’s  quarterly  monitoring 
visits,  and  remained  undetected  until  the  renewal  evaluation.  Problems 
not  detected  included  data-system  inadequacies,  failure  to  implement 
interventions  for  admissions  and  quality  objectives  in  a  timely  manner, 
inadequate  internal  control  measures  to  check  on  the  performance  of 
nurse-reviewers  and  physicians,  and  failure  to  act  on  detected  quality 
issues  not  included  in  objectives. 

For  example,  hcfa  regional  office  monitoring  personnel  failed  to  detect 
serious  problems  with  the  data  system  of  one  of  the  six  before  the 
renewal  evaluation.  When  verifying  the  data  supplied  by  this  pro  on  the 
self-evaluation  section  of  the  evaluation  document,  hcfa  regional  office 
personnel  discovered  that  the  pro’s  data  system  did  not  include  data 
from  its  two  review  subcontractors.  As  a  result,  hcfa  regional  office  per¬ 
sonnel  were  unable  to  verify  the  accuracy  of  the  numbers  supplied  by 
the  PRO,  including  those  it  reported  for  its  admission  and  quality  objec¬ 
tives.  Furthermore,  the  data  system  was  so  inadequate  that  the  pro  had 


HCFA  Monitoring  Did 
Not  Always  Detect 
Poor  PRO  Performance 


Page  26 


GAO/HRIV88-13  PRO  Evaluations 


Chapters 

Routine  PRO  Monitoring  Could  Be  Improved 
to  Detect  and  Correct  Poor  Performance 


to  do  its  profiling  manually,  a  task  that  hcfa  officials  considered  virtu¬ 
ally  impossible  in  a  large  state.  Although  this  inadequate  data  system 
had  been  the  source  of  the  reports  that  the  pro  was  required  to  provide 
HCFA,  we  could  find  no  indication,  in  either  prompts  or  the  regional  office 
correspondence  files  with  this  pro,  that  hcfa  monitoring  personnel  had 
previously  noticed  this  problem. 

Under  their  contracts  PROS  were  required  to  take  actions — called  inter¬ 
ventions — to  correct  problem  areas  identified  in  the  contract.  For  exam¬ 
ple,  one  PRO  we  reviewed  contracted  to  reduce  avoidable  postoperative 
or  other  complications.  As  part  of  this  objective,  it  agreed  in  its  contract 
to  take  specific  actions  intended  to  achieve  this  objective  at  specific 
times.  For  instance,  the  pro  agreed  that  in  October  1984,  it  would  begin 
to  review  all  cases  of  providers  with  excessive  numbers  of  patients  who 
had  complications  resulting  from  hospitalization.  However,  the  PRO  did 
not  begin  this  activity  until  September  1986.  In  another  instance,  the  pro 
agreed  to  require  providers  that  had  not  corrected  their  problems  by 
December  1984  to  begin  education  programs.  However,  it  had  not  yet 
begun  this  activity  at  the  time  the  pro  prepared  its  self-evaluation  in 
January  1986.  The  pro  received  a  score  of  zero  in  the  quality  objectives 
interventions  element  of  its  evaluation. 

PROS  for  five  of  the  six  states  whose  evaluations  we  reviewed  performed 
poorly  in  the  area  of  interventions  for  their  quality  or  admissions  objec¬ 
tives.  In  general,  this  occurred  because  the  pro  did  not  intervene  until 
well  after  the  times  required  under  the  contract.  Although  prompts  did 
include  questions  on  the  timeliness  of  interventions,  this  had  not  been 
identified  as  a  significant  problem  with  four  of  these  pros. 

Four  of  the  six  pros  performed  poorly  in  the  area  of  correction  of 
quality-of-care  problems  not  covered  by  the  quality  objectives.  In  three 
of  the  four  cases,  there  was  no  indication  that  this  problem  had  been 
identified  before  the  renewal  evaluation,  which  could  have  resulted  in 
inadequate  protection  of  Medicare  beneficiaries  against  poor  quality  of 
care  because  identified  quality  problems  were  not  corrected. 

Two  PROS  received  low  scores  because  they  had  inadequate  internal  con¬ 
trol  systems.  These  systems  are  designed  to  measure  and  correct  defi¬ 
ciencies  in  the  performance  of  pro  personnel,  primarily  nurse-reviewers, 
who  perform  the  initial  review  of  the  medical  records,  and  physicians, 
who  decide  whether  problems  referred  to  them  by  the  nurse-reviewers 
are  significant.  Despite  the  importance  of  having  such  a  system,  there 
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was  no  evidence  that  this  deficiency  had  bet*n  identified  in  thest>  two 
PROS  before  the  renewal  evaluation. 

Failure  to  detect  and  correct  pro  performance  deficiencies  until  the  con¬ 
tract  renewal  evaluation,  three-quarters  of  the  way  into  the  contract 
period,  was  a  serious  deficiency  in  mcfa’s  management  of  the  program. 
The  contracts  were  intended  to  carry  out  the  intent  of  the  pro  program 
authorizing  legislation  to  protect  the  Medicare  program  and  benefi¬ 
ciaries  against  unnecessary  utilization  and  poor  quality  of  care.  To  the 
extent  that  ucfa  had  not  identified  pro  performance  problems,  hcfa  had 
not  assured  that  the  program  was  accomplishing  its  functions. 

HCFA  officials  were  aware  of  this  deficiency  when  we  began  our  review. 
They  redesigned  the  prompts  system  (prompts-2)  to  correct  this  problem 
in  the  1986-88  contract  period.'  They  also  said  promi>ts-2  will  form  the 
basis  of  the  contract  renewal  evaluations  to  be  conducted  in  1988.  How¬ 
ever,  prompts-2  was  not  available  until  February  27,  1987,  about  7 
months  after  the  effective  date  of  the  first  of  the  current  round  of  con¬ 
tracts.  In  addition,  as  of  March  31,  1987,  hcfa  had  not  decided  how 
information  not  covered  in  prompts-2  would  bo  included  in  the  renewal 
evaluation  or  how  this  evaluation  would  be  conducted. 


A  major  reason  for  hcfa  monitoring  personnel  not  identifying  significant 
PRO  problems  until  the  renewal  evaluation  was  that  the  principal  tool 
used  by  hcfa  regional  office  personnel  for  routine  monitoring  of  the 
PROS — PROMPTS — did  not  adequately  address  performance  in  many  ele¬ 
ments  stressed  in  the  renewal  evaluation. 

For  example,  the  pro  Evaluation  Protocol  required  that  the  pro  list  each 
potential  sanctionable  problem,  with  the  date  on  which  the  pro  notified 
the  hospital  or  physician,  and  its  current  status.  However,  promits 
asked  only  if  the  pro  was  properly  implementing  its  sanction  plan  and 
properly  reporting  its  sanction  activity  to  hcfa.  promi’TS  did  not  require 
the  monitoring  official  to  assure  that  sanctionable  i.ssues,  once  identi¬ 
fied,  were  being  promptly  resolved. 

PROMPTS  also  did  not  contain  questions  on  the  issue  of  the  adequacy  of 
the  pros’  data  systems.  Nor  did  it  encourage  hcfa  regional  office  person¬ 
nel  charged  which  monitoring  the  pros’  activities  to  investigate  the  accu¬ 
racy  of  the  data  reported  by  the  F’ROs.  Rather,  promits  simply  asked 


'The  contracts  cover  a  2-year  p»'ri(xl  betw(S‘ii  roughly  mid-l.'ISli  through  mid  lilfW 
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them  to  determine  whether  the  rept)rt.s  were  rweived  on  time  and  were 
ac-ceptable.  Staff  at  one  regional  office  we  visited  told  us  that  as  long  as 
the  data  reports  were  internally  consistent  and  not  obviously  erroneous 
they  would  not  check  the  validity  of  the  data  behind  the  reports  While 
there  was  no  element  of  the  renewal  evaluation  spe<‘ifically  directtKl  at 
the  adequacy  of  the  mi's  data  system,  in  the  cast'  of  one  t'Ko,  ut  r.A 
regional  office  monitoring  personnel  discovered  that  the  I’Ko  had  the 
data  system  problem  discu.s.st'd  on  pagt*s  2H-27  when  they  found  that 
they  were  unable  to  verify  much  of  the  data  supplied  by  the  j*ho  in  its 
self-evaluation  report. 


Quality-of-Care  Issues  Not 
Stressed  in  PROMPTS 


Four  of  the  six  pros  we  visited  were  found  to  b«>  deficient  in  the  quality- 
of-care  element  because  they  did  not  properly  follow  up  on  identified 
quality-of-care  problems.  Although  pro  activitit's  in  identifying  and  cor¬ 
recting  qualitynif-care  problems  in  areas  not  covered  by  their  quality 
objectives  were  an  important  element  in  the  renewal  evaluation,  the 
prompts  dcK-ument  did  not  contain  qiu*stions  pertaining  to  the  resolution 
of  .such  problems. 

Quality-of-care  issues  are  identified  in  the  courst'  of  performing  medical 
review  activities.  The  Medical  Review  section  of  promits  contained 
questions  that  would  have  indicated  if  the  pro  was  finding  quality-of- 
care  problems  in  the  cases  it  reviewed.  However,  the  Medical  Review 
section  did  not  contain  questions  addressing  the  more  important  i.ssue  of 
what  the  pro  did  to  correct  the  problenvs  identified.  SuperPRo  reviews 
also  identified  quality-of-care  issues  that  the  pro  did  not  find,  but  did 
not  address  the  resolution  of  those  problems  that  the  pro  had  found. 
Also,  none  of  the  data  repxuts  supplied  by  the  pro  containt'd  information 
on  PRO  activities  in  this  area. 


H 


In  contrast,  the  contract  renewal  evaluations  contained  two  elements 
that  addressed  identification  of  quality-of-care  issues  not  containiHi  in 
PRO  objectives.  One  element  required  the  pro  to  list  each  quality-of-care 
problem  it  had  identified  in  its  medical  review  activities,  the  date  identi¬ 
fied.  actions  taken  to  resolve  it.  and  the  results  or  jx*nding  ac  tions  if  it 
was  still  unresolved.  In  addition,  the  pro  was  rc'quirc'd  to  give  similar 
information  on  cases  that  had  been  referred  to  it  by  other  partu's. 
including  Membc*rs  of  Congress,  the  nils  Office  (»f  the'  insjxvtor  General, 
and  HCKA. 

The  monitoring  .system  may  not  have*  stres.s<‘d  the*  area  of  pro  activit>’  in 
correcting  quality-of-care  problems  outside  its  quality  objes  tives  in  part 
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betause  of  what  appears  to  have  bt*en  a  )?reater  emphasis  on  utilization  I 
control  in  the  early  part  of  the  contract  periotl  Neither  PKoMrrs  nor  the 
scope  of  work  for  the  1984  contract  dealt  very  extensively  with  quality 
of  care  when  compart'd  with  the  amoun^  of  attention  given  to  the  issue 
of  detecting  and  correcting  overutilization  problems,  such  as  uniUH-es- 
sary  admissions.  However,  by  the  time  of  the  evaluation,  emphasis 
seems  to  have  shifted  toward  quality-of-care  issues. 

Some  HfKA  officials  we  talked  to  acknowledged  this  shift  in  emphasis. 

For  example,  one  im  ka  official  involved  in  the  development  of  I’Komits 
told  us  that  the  differences  in  emphasis  on  quality  between  I'Komits  and 
the  contract  evaluations  reflected  the  fact  that  there  were  different  con¬ 
cerns  in  1984,  when  pkomits  was  developed,  and  1986,  when  the  evalua¬ 
tions  were  performed.  Another  mcka  official  involved  in  monitoring  pro 
performance  said  that  there  may  have  bei*n  a  shift  in  monitoring  empha¬ 
sis  toward  quality  in  the  middle  of  the  contract  periixl.  although  there 
was  no  change  in  the  contract  requirement  to  identify  and  correct  qual¬ 
ity  problems.  Another  hcka  official  involved  in  pro  monitoring  told  us 
that  monitoring  nuos  in  the  area  of  quality  was  not  emphasized  until  the 
second  half  of  the  c-ontract  period. 

While  we  believe  that  IK  ka’s  emphasis  on  assunng  that  pros  detect  and 
correct  quality-of-care  problems  whether  or  not  they  form  part  of  the 
pro’s  objectives  is  c*orrect,  ucka's  apparent  increase  in  emphasis  on 
quality-of-care  issues  during  the  contract  period  did  pose  a  problem  for 
PROS.  Because  hcka’s  monitoring  program  was  deficient  in  this  area,  it 
was  less  likely  that  either  hcka  or  the  pros  would  be  aware  that  the  pr(»s 
were  performing  inadintuately  in  this  area  bt'fore  it  was  reveaUnl  by  the 
renewal  evaluation. 


PROMPTS-2  Covers 
Most  Weak  Areas  in 
PROMPTS  but 
Sometimes  Lacks 
Criteria 


The  new  prompts-2  should  better  fiKHs  the  attention  of  hcka  monitor  ing 
personnel  on  those  areas  of  contract  performance  in  which  problems 
remained  undetwted  during  the  previous  contract.  However,  in  our 
analysis  of  prompts-2.  we  found  some  instances  where  it  lacks  clear  cri¬ 
teria  to  assist  HiTA  monitoring  personnel  in  determining  exactly  what 
constituU's  acceptable  performance 

promI’TS-2  addres.s<*s  most  areas  in  which  we  found  the  original  PRoMri's 
to  bt'  deficient  For  example,  while  promi''I>  nsjuired  mcka  monitoring 
personnel  to  determine  if  the  pro  was  prcMlucing  profiles,  it  did  not  rais«“ 
the  issue  of  how  the  pro  was  using  them  PRoMri's  2  requires  monitoring 
personnel  to  assun*  that  the  pro  is  using  profiles  to  identify  problems 
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Again,  while  pkomits  did  not  ri*quire  iicka  monitoring  {X'rsonnel  to 
examine  the  aeeuraey  of  the  data  reported  by  the  I'Ho.  I’KoMnN-li 
requires  that  UO’A  monitoring  personnel  assure  themselves  that  thes<* 
data  are  accurate.  In  the  area  of  quality  issues  not  covered  by  pko  qual 
ity  objectives,  an  area  where  many  I’Kos  had  problems  identified 
through  the  evaluation,  I'Homi’TS  had  no  questions  at  all.  how¬ 

ever,  requires  hcka  monitoring  personnel  to  assure  themselv<*s  that  the 
PRO  acts  promptly  to  resolve  quality  issiu*s  and  to  prevent  their 
recurrence. 

Although  pH()MPTS-2  reprt»sent.s  an  improvement  over  pkomi’Ts,  our  anal¬ 
ysis  found  a  potential  problem.  In  some  areas  PKOMnv2  lacks  criteria  to 
help  HCKA  monitoring  personnel  determine  the  difference  between 
acceptable  and  unacceptable  performance.  For  example,  in  the  area  of 
interventions  to  corrt*ct  quality  problems,  I'Kompts-2  asks  the  monitor  to 
determine  if  the  pro  has  effectively  identified  areas  needing  interven¬ 
tion,  promptly  resolved  the  problems,  and  addressed  them  appropri¬ 
ately.  Nowhere  does  it  define  what  is  meant  by  “effectively," 
“promptly,"  or  "appropriately,”  In  another  instance,  the  hcka  mtinitor  is 
asked  to  determine  if  the  pro’s  internal  control  system  is  adequate.  How¬ 
ever,  it  does  not  indicate  what  constituU*s  an  "adequate  ”  internal  con¬ 
trol  system. 

The  HCKA  monitors  are  also  asked  to  judge  if  the  relationships  betw'een 
the  PRO  and  the  intermediary,  and  betwi*en  the  pro  and  providers,  are 
"working  effectively”  They  are  asktnl  to  list  stx*cific  problems  The  d<K - 
ument  contains  no  definition  of  what  constitutes  an  effective  working 
relationship  or  what  constitutes  a  problem  in  working  relationships. 

In  these  instances,  hcka  is  depending  on  the  individual  judgment  of  its 
monitoring  personnel  to  determine  what  constitutes  acceptable  |x*rform- 
ance.  This  could  lead  to  inconsistent  rt'sults  w  hen  pr<)MI>ts-2  is  appluxl  to 
different  pros. 

When  we  discus.sed  this  with  hcka  officials,  they  did  not  agrw  that  it 
was  significant.  They  said  that  in  some  cases,  such  criteria  are  found  in 
other  dwuments.  In  addition,  they  argued  that  hcka  regional  office  mon¬ 
itoring  personnel  had  sufficient  expedience  with  the  program,  and  suffi¬ 
cient  knowledge  of  the  history  of  the  individual  pros  for  which  they 
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were  responsible,  to  make  amirate  judXment.s  regarding  what  is  areept 
able  and  what  is  not  They  nou*d  that  hcka  int<*nded  to  conduct  training 
for  monitoring  personnel  and  was  reviewing  all  eomH  tivi-  action  plans 
to  assure  that  decisions  made  by  monitoring  |)t>rsonnel  were  fair  and 
cxmsistent.  However,  we  believe  that  inconsistent  decisions  by  lu  ka  mon 
itoring  personnel  would  be  less  likely  and  identification  of  performance 
problems  more  as.sured  if  criteria  were  included  in  I'Romits  ^ 


Conclusions 


HCKA  did  not  detec-t  and  correct  many  instances  of  p<H)r  I’Ko  |x*rformanc  e 
before  the  contract  renewal  evaluation.  This  meant  that  many  I’Kos  were 
not  adequately  carrying  out  import iint  rt*quirements  of  their  contracts 
for  most  of  their  contract  periods  and  thus  the  Medicare  program  and  its 
beneficiaries  may  not  have  received  all  the  protection  intended 

At  the  time  we  began  our  audit  work,  iicka  program  officials  told  us  that 
the  number  of  pkos  failing  the  contract  renewal  evaluation  had  alerttxl 
them  to  the  need  to  improve  their  pro  monitoring  efforts  and  that  they 
were  taking  action  to  do  so.  They  said  they  had  redesigned  pkomits  to 
better  focus  regional  office  monitoring  efforts  on  important  measures  of 
outcome  and  they  intended  to  use  it  as  the  basis  for  the  next  renewal 
evaluation. 

We  believe  that  prompts-2  is  a  significant  improvement  over  I'Rompts  It 
covers  most  of  the  areas  in  which  prompts  was  deficient.  However,  in 
some  cases,  it  may  not  provide  adequate  criteria  to  enable  iicka  monitor¬ 
ing'.  personnel  to  differentiate  between  acceptable  and  unacceptable  jxt- 
formance.  We  believe  that  to  assure  greater  consistency  in  evaluating 
PRO  performance,  such  criteria  .should  be  included  in  the  d»Kument, 
either  directly  or  by  reference  if  the  criteria  are  contained  in  other 
documents. 


Recommendation 


We  recommend  that  the  Secretary  of  mis  direct  the  Admini.strator  of 
HCKA  to  provide  criteria  to  enable  iicka  pt'rsonnel  to  differentiate 
between  acceptable  and  unacceptable  performance  in  the  routine  moni¬ 
toring  of  pros’  activities. 


Agency  Conunents  and 
Our  Evaluation 


The  Hits  inspector  general  stated  that  Hits  generally  agreed  with  this  ris  - 
ommendation  and  indicated  that,  where  appropriate,  lu  ka  would  con¬ 
tinue  to  provide  criteria  to  its  personnel  for  differentiating  Ix-tween 
acc«*ptable  and  unacceptable  jx'rformance  He  said  that  Hi  ka  regional 
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office  [K‘rsonn«*l  have  Hiven  criteria  outside  of  ttic  l’Ki»Mn>  do<  ii 
ment  b«*<‘aus«*  the  diM-iiment  is  not  inteiuhnl  to  s«*rve  as  an  instructional 
devur  or  manual  for  proj<*ct  officers  He  also  said  that  trainiii)?  pro 
t(rams  have  btvn  held  and  instructional  mati'rials  pro\  ided  He  also  said 
IK  Ka  believ»‘s  that  criteria  containtHl  in  the  individual  I’Ko  contracts.  i*Ko 
instructional  issuanc«*s,  and  the  cko  monitoring  protiK-ols.  as  well  as  the 
training;  and  instnictional  materials,  will  n^siilt  in  appropriate  monitor 
ing  of  i*Ko  j)erf»»rmance 

We  aj<rts‘  that  adi‘quate  criteria  may  la-  availabU‘  m  other  diK  uments 
However,  we  continue  to  Ixdieve  that  incorporating  such  criteria  in  the 
miMI’Ts-2  d<K-ument.  either  diris  tly  or  by  reference,  would  help  assure 
that  they  would  be  uniformly  and  consistently  applit^  by  IK  fa's  i*ko 
monitoring  pt*rsonnel 

AMPKA  agreed  with  our  findings  and  this  nsommendation  a.mI’Ra  slau*d, 
that  deficiencu‘s  in  pro o|)erat ions  should  la*  faiinted  out  to  the  pro  as 
early  in  the  contract  cycle  as  possible  ampka  said  waiting  for  the  final 
evaluation  before  defu  iencit's  are  identifietl  is  not  fair  to  the  pro,  which 
has  assumed  hcfa  is  satisfUnl  with  operations  until  that  point,  and  does 
not  permit  time  for  corrtH  tive  action 

AMPKA  also  said  that  PKOMiTS-’i  is  an  improvement  over  earlier  evalua¬ 
tion  instniments  ampra  expn*ss<*d  its  concern,  however,  that  phomits-2 
lacks  spei  ific  criteria  to  measure  acceptable  and  unacceptable  |x*rform- 
anev  and  that  w’ithout  defim*d  criteria,  the  subje<  tive  judgments  of  dif¬ 
ferent  MCFA  personnel  will  continue  to  influence  final  evaluations  of 
PROS 
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MC  KA  designed  the  seope  of  the  198()-H8  I’Ko  eontracLs  around  the  mini-  ^ 

mum  level  of  funding  pi'rmitted  by  law  Furthermore,  bec  ause  of  uncer¬ 
tainties  in  the  methodology  hcka  used  to  determine  the  funding  level  for 
each  HKC)  contract,  we  do  not  know  whether  the  t’Kos  are  appropriately 
funded  to  perform  the  level  of  effort  included  in  the  contracts.  Wc> 
believe  that  UffA  should  initially  dc'termine  the  sc-opt*  of  the  I’Ko  con- 
trac-ts  netnlc^  to  protec-t  the  Medicare  program  and  its  bc*neficiaru*s  and 
use*  this  as  the  starting  point  for  determining  the  program's  funding 
level. 


PRO  Funding  for  1986- 
88  Contracts  Based  on 
Minimum  Required  by 
Law 


The  funding  for  individual  t’Ko  contracts  for  the  current  2-year  contrac  ts 
IS  based  on  the  minimum  program  funding  permitted  by  law  In  the 
Social  Security  Amendments  of  1983,  the  Congress  required  that  the  pko 
program  be  funded  directly  from  the  Hospital  Insurance  Tnist  PTind  and 
set  a  minimum  funding  level.  It  required  that  Medicare  inpatient  hospi¬ 
tal  review  by  pros  be  funded  at  an  amount  not  less  than  the  tost  per 
review  set  in  fiscal  year  1982,  and  not  less  in  the  aggregate  than  the 
amount  expended  in  fiscal  year  1982  for  the  i’sro  program,  when 
acUusted  for  inflation.  Under  this  provision,  ucka  based  the  cost  of  the 
program  on  the  cost  per  admission  in  1982,  as  at^justed  for  inflation. 
After  UCKA  established  the  budget  target  for  the  I’RO  program,  the  Con¬ 
solidated  Omnibus  Budget  Reconciliation  Act  of  1985  (P.L.  99-272)  mod¬ 
ified  the  flcKir  so  that  funding  cannot  be  less  than  the  amount  expended 
in  fiscal  year  1986,  ac^usted  for  inflation. 


MC  KA  essentially  considered  the  1983  act’s  budgetary  “floor”  to  be  the 
■’c-eiling"  as  well,  and  all  budget  estimates  for  the  contracts  awardei^’  ,n 
1 986  were  made  based  on  the  minimum  budget  allowed  by  law.  Accord¬ 
ing  to  a  MCKA  official,  these  estimates  were  made  before  the  scope  of 
work  was  developed  and  were  revised  during  the  budgeting  process 
bevause  of  changes  in  the  actuarial  estimates  of  the  number  of  Medicare 
hospital  admissions  during  the  contract  period.  Initially,  the  hcfa  actua¬ 
ries  had  assumed  that  the  numbers  of  admissions  would  continue  to 
increase  under  Pi>s  as  they  had  under  cost  reimbursement.  However,  as 
data  about  Medicare  admissions  under  pps  began  to  come  in,  hcfa  real¬ 
ized  that  admissions  were  not  increasing  as  expected,  hcfa  officials, 
therefore,  decided  to  base  the  budget  estimates  on  the  assumption  that 
the  numbers  of  admissions  would  remain  constant  at  the  1984  level. 

This  pnx-ess  resulted  in  a  final  budget  estimate  for  fiscal  years  1987  and 
1988.  es.sentially  the  contract  period  for  those  contracts  signed  in  1986, 
of  about  $357  million  for  inpatient  hospital  reviews. 
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According  to  a  hcfa  official  involved  in  the  pko  budgeting  process,  hcfa 
aimed  at  an  initial  contract  figure  of  about  $320  million,  leaving  a 
reserve  of  $37  million,  or  more  than  10  percent  of  the  total  amount,  to 
pay  for  unexpected  expenses,  such  as  contract  terminations,  and  for 
pros'  activities  in  preparing  sanction  cases,  which  are  separately  funded 
through  contract  mcxlifications.  However,  we  estimate  the  total  funding 
for  the  1986-88  contract  cycle  at  about  $293  million.  Thus,  hcfa  has  a 
program  reserve  of  about  $64  million,  or  almost  18  percent,  rather  than 
the  target  10  percent. 

This  raises  the  possibility  that  hcfa  may  not  be  able  to  spend  up  to  the 
legislative  budgetary  floor  during  this  2-year  contract  cycle.  We  asked  a 
HCFA  official  responsible  for  pro  budgeting  if  this  was  a  possibility.  He 
responded  that  underspending  the  floor  amount  was  unlikely,  because 
the  floor  had  been  changed  to  the  1986  program  costs,  updated  for  infla¬ 
tion.  According  to  the  figures  contained  in  the  President’s  1988  budget, 
the  program  cost  $151  million  in  1986,  which  corresponds  to  a  spending 
floor  for  1986-88  of  about  $318  million,  using  the  most  recent  inflation 
estimates. 


Scope  of  Work 
Designed  to  Keep 
Program  Funding 
Within  Minimum 


The  scop)e  of  work  for  the  1986-88  contracts  was  designed  to  keep  the 
cost  of  the  program  at  the  legal  minimum.  In  addition,  there  were  so 
many  uncertainties  in  the  methodologies  hcfa  used  to  estimate  the  cost 
of  performing  the  scope  of  work  that  was  established  that  we  cannot  be 
certain  that  hcfa’s  funding  estimates  for  individual  pros  reflect  the 
actual  costs  of  performing  that  work,  hcfa  program  officials  generally 
agreed  with  our  findings  and  told  us  that  these  estimates  were  intended 
as  general  guides  to  the  contract  officers  in  negotiating  the  prices  of  the 
contracts. 


Scopie  of  Work  Designed  hcfa  officials,  when  designing  the  scope  of  work  for  the  1986  contracts. 
Around  Minimum  Funding  decided  that  they  would  have  to  work  within  a  similar  amount  of  money 

as  in  the  previous  contract  cycle.  However,  based  on  their  experience 
during  the  first  contracts,  hcfa  decided  to  increase  the  intensity  of  case 
reviews,  primarily  to  review  more  cases  for  quality  problems.  This 
would  have  the  effect  of  increasing  the  average  cost  per  review.  Hecause 
HCFA  was  attempting  to  keep  the  cost  of  the  program  constant,  the 
increased  review  c().st.s  had  to  be  offset  by  reducing  the  number  of  cases 
reviewed. 
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Tabto  4.1:  Parcantage  of  Total 
Admiaaiona  to  Ba  Raviawad,  by 
Catagory 


According  to  hcfa  officials,  the  scope  of  work  for  the  first  contract  cycle 
was  designed  so  that  pros  would  review  about  30  percent  of  Medicare 
inpatient  admissions.  (In  fact,  overall,  pros  reviewed  40  percent  or 
more.)  In  preparing  the  scope  of  work  for  the  1986  contract  cycle,  hcfa 
reduced  the  estimated  percentage  of  admissions  to  be  reviewed  to  about 
26  percent.  Table  4. 1  gives  the  breakdown  of  this  percentage  into 
categories. 


Review  category 

Percentage  of  total 
admissions 

0  066 

All  cases  with  specified  diagnosis  codes 

Hospital  notices  of  noncoverage 

0  456 

Day  or  cost  outliers 

1  500 

Transfers  to  other  hospitals  or  to  PPS  exempt  units 

1  135 

Readmissions  of  patients  within  15  days  of  discharge® 

3000 

Validation  of  PPS  payment  category  assigned  to  admission 

2960 

Preadmission  reviews 

2500 

Specialty  hospitals 

1  000 

Random  sample  (includes  intensified  review  when  problems  found) 

9890 

Validation  of  contracted  objectives 

0310 

Contracted  objectives 

3.000 

Total 

25.817 

“The  Omnibus  Budget  Reconciliation  Act  ol  1986  requires  HCFA  to  require  PROs  to  review  at  least  a 
sample  of  all  hospital  readmissions  within  31  days  of  a  prior  discharge  However,  this  provision  is  effec¬ 
tive  only  for  contracts  entered  into  or  renewed  on  or  after  January  1 .  1987  All  of  the  contracts  for  the 
current  2-year  period  were  effective  before  this  dale 


Uncertainties  in  Scopie  of  According  to  a  hcfa  official  who  helped  develop  this  estimate,  the  actual 

percentage  of  cases  that  each  pro  will  have  to  review  within  each  of  the 
individual  components  varies.  The  3-percent  random  sample'  of  all  cases 
was  set  by  hcfa.  Some  others,  such  as  those  for  outliers  and  validation 
of  PPS  payment  category,  were  based  ori  historical  data  from  Medicare’s 
statistical  information  systems  and  are,  therefore,  relatively  reliable. 
Others  are  based  on  less  "hard”  information.  For  example,  this  official 
pointed  out  that  the  estimate  for  readmissions  is  subject  to  uncertainty 
because  of  the  need  to  apply  Judgment  in  determining  which  readmis¬ 
sions  were  related  to  the  medical  conditions  of  a  prior  admission.  In 
addition,  he  said  that  the  3-percent  estimate  for  contract  objectives  is 


'The  9.89  per«‘nt  in  Wble  4  I  ineliides  an  additional  cstimaled  l>  89  ptTa'nl  for  intensifierl  n'view 
stemming  from  errors  found  m  the  .'t-iieraml  random  sample 
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Uncertainties  in  HCFA’s 
Contract  Price  Estimates 


uncertain  because  pros  have  contracted  for  different  objectives.  Fur¬ 
thermore,  according  to  this  official,  all  of  these  estimates  were  national 
in  scope,  and  individual  pros  could  differ  from  the  estimates  because  of 
local  differences. 


HCFA  used  several  assumptions  to  dev'elop  its  initial  cost  estimates  for 
PRO  contracts  that  result  in  a  degree  of  uncertainty  about  these  esti¬ 
mates.  HCFA  officials  told  us  that  the  estimates  were  intended  to  be  no 
more  than  a  general  guide  for  contract  negotiators.  We  found  that  the 
actual  percentages  of  cases  reviewed  and  costs  per  case  in  awarded  con¬ 
tracts  varied  significantly  from  those  estimated. 

As  part  of  their  preparation  for  negotiating  the  1986  round  of  contracts, 
HCFA  prepared  state-by-state  estimates  of  what  it  should  cost  to  perform 
each  review  category  shown  in  table  4. 1 .  According  to  the  hcfa  official 
responsible  for  preparing  these  estimates,  hcfa  staff  began  by  develop¬ 
ing  direct  labor  costs  for  each  review  category.  To  do  this,  they  devel¬ 
oped  estimates  of  the  time  that  would  be  required  to  perform  each  type 
of  review  by  registered  nurses  and  other  specialists  using  hcfa  person¬ 
nel  in  these  professions.  They  then  estimated  wage  rates  for  nurse 
reviewers  from  Bureau  of  Labor  Statistics  hourly  wage  rates  for  the  spe¬ 
cific  areas,  adjusted  for  inflation.  Wage  rates  for  accredited  medical 
records  technicians  were  developed  based  on  a  1983  American  Medical 
Records  Association  survey  of  its  membership.  The  percentage  of  cases 
requiring  physician  involvement  and  the  amount  of  physician  time  per 
review  was  estimated  based  on  the  experience  of  hcfa’s  in-house  physi¬ 
cians.  For  physician  wage  rates,  hcfa  used  a  national  rate  of  $54  per 
hour.  According  to  this  official,  hcfa  did  not  attempt  to  use  actual  PRO 
experience  in  estimating  nurse  or  physician  cost  per  review  because  it 
did  not  have  reliable  pro  cost  information. 

Using  a  model  based  an  the  actual  experience  of  a  random  sample  of  21 
I’ROs,  HCFA  used  these  direct  labor  costs  as  the  basis  for  estimating  the 
other  costs  of  a  pro,  such  as  salaries  of  administrative  personnel,  fringe 
beneots,  overhead,  and  subcontracting  costs.  In  addition,  if  a  pro  had 
one  or  more  contracts  to  perform  review  for  other  governmental  or  non¬ 
governmental  entities,  hcfa  reduced  its  estimate  by  prorating  the  pro’s 
overhead  costs  between  the  pro’s  other  business  and  its  Medicare 
contract. 

This  calculation  gave  hcfa  an  estimated  cost  per  case  for  performing 
each  type  of  review.  Overall  costs  for  each  pro  were  then  computed  by 
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multiplying  this  figure  times  the  number  of  reviews  of  each  type  for 
each  state.  The  latter  figure  was  obtained  by  multiplying  the  nationally 
estimated  percentage  of  reviews  of  each  type  times  the  estimated  total 
number  of  admissions  for  each  state.  This  latter  number  was  in  turn 
developed  by  combining  actual  experience  data  by  state  for  an  earlier 
year  with  national  actuarial  estimates.  According  to  a  hcfa  official,  this 
was  done  because  ucka  did  not  have  the  numbers  of  admissions  per  state 
on  a  current  basis. 

This  HCFA  official  stated  that  these  estimates  were  only  intended  to  be 
used  as  a  starting  point  for  negotiations  with  the  pro  by  the  contracting 
officer.  They  did  not  take  into  account  differences  in  patterns  of  health 
care  among  the  states,  which  could  cause  the  actual  level  of  effort 
needed  by  pros  to  differ  considerably.  Thus,  it  was  expected  that  the 
actual  amounts  awarded  to  a  pro  might  differ  significantly  from  the  esti¬ 
mate  developed  using  the  model. 

The  national  actuarial  estimates  of  admissions,  which  as  noted  above 
had  already  been  revised  once  to  assume  no  increase  in  Medicare  admis¬ 
sions,  were  again  revised  downward  to  assume  a  decreasing  number  of 
Medicare  admissions.  According  to  the  hcfa  official  in  charge  of  the  f»ro 
budgeting  process,  this  reduction  in  the  estimated  number  of  admissions 
was  not  used  to  recalculate  the  total  amount  available  for  the  pro  pro¬ 
gram.  However,  hcfa  required  all  but  a  few  pros  with  the  earliest 
awarded  contracts  to  base  their  offered  price  estimates  on  the  revised, 
lower  admission  estimates.  In  the  instance  where  we  observed  a  negotia¬ 
tion,  the  PRO,  which  had  been  instructed  to  use  the  earlier  admission 
estimates  as  the  basis  for  its  offer,  was  told  that  it  would  have  to  base 
its  offer  on  the  later,  lower  estimates  when  it  arrived  for  the  negotiating 
session. 

HCFA  officials  told  us  that  they  were  prepared  to  give  pros  additional 
funding  if  they  could  show  that  they  would  do  additional  reviews  above 
the  estimated  percentages.  In  fact,  for  most  of  the  contracts  for  the 
1986-88  contract  cycle,  the  negotiated  percentage  of  admissions  to  be 
reviewed  was  above  the  25.8-percent  hcfa  national  estimate  and  led  to 
increases  in  contract  prices.  In  the  instance  where  we  observed  the  con¬ 
tract  negotiation  process,  hcfa  negotiators  increased  the  price  by  about 
$2  million  when  the  pro  was  able  to  document  that  it  would  be  doing 
more  reviews  in  certain  categories  (for  a  total  of  27.6  per  cent  of  all 
admissions)  than  had  been  estimated  (25.8  percent). 
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Furthermore,  for  most  I’ROs  the  contract  cost  per  case  reviewed  differed 
significantly  from  the  original  iiCFA  estimates.  However,  in  contrast  to 
the  percentage  of  reviews,  nearly  as  many  final  contract  prices  were 
below  the  estimated  cost  as  were  above  it,  and  the  overall  average  cost 
per  review  was  less  than  2  percent  above  the  estimated  cost. 


Conclusions  designed  the  scope  of  work  of  the  1986-88  contracts  to  keep  the 

cost  of  the  program  close  to  the  legislatively  mandated  minimum  fund¬ 
ing  level.  If  this  funding  level  is  inadequate,  the  Medicare  program  and 
its  beneficiaries  would  not  receive  the  degree  of  protection  from  unnec¬ 
essary  and  poor  quality  hospital  care  the  Congress  intended  the  pro  pro¬ 
gram  to  provide. 

There  were  many  uncertainties  in  the  methodology  that  ucfa  u.sed  to 
estimate  the  costs  to  individual  pros  in  carrying  out  the  scope  of  work  of 
the  contract.  As  a  result,  we  could  not  determine  whether  hcfa’s  cost 
estimates  were  reasonable  estimates  of  the  costs  to  the  pros  of  carrying 
out  the  required  work.  However,  the  fact  that  the  contract  awards  differ 
from  the  estimated  costs  in  both  cost  per  review  and  number  of  reviews 
suggests  that  the  contract  negotiation  process  may  have,  at  least  in  part, 
compensated  for  these  uncertainties. 

In  the  final  analysis,  if  contract  prices  are  too  low,  hcfa’s  willingness 
and  ability  to  make  modifications  to  contract  prices  will  be  the  determi¬ 
nant  of  whether  pros  receive  sufficient  funding  to  carry  out  the  scope  of 
work. 


R6C0inmen(la.ti0nS  recommend  that  the  Secretary  of  hhs  direct  the  Administrator  of 

HCFA  to 

•  determine  the  scope  of  review  needed  to  adequately  meet  the  program’s 
intent  and  use  this  as  the  starting  point  for  determining  the  program’s 
funding  level  and 

•  collect  and  use  adequate  cost  and  performance  data  to  set  each  r’RO's 
contract  funding  at  a  level  sufficient  to  provide  the  coverage  determined 
to  be  necessary. 
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Agency  Comments  and 
Our  Evaluation 


The  inspector  general  said  that  hhs  concurred  with  our  recommendation 
that  HCFA  should  determine  the  scope  of  review  needed  to  adequately 
meet  the  program’s  intent  and  use  this  as  the  starting  point  for  deter¬ 
mining  the  program’s  funding  level.  He  said,  however,  that  this  recom¬ 
mendation  stemmed  from  our  misunderstanding  of  how  the  scope  of 
review  was  developed.  According  to  the  inspector  general,  the  scope  of 
work  supporting  PRO  contracts  was  developed  based  upon  experience 
gained  from  medical  review  by  psros  and  fiscal  intermediaries  and  was 
designed  to  address  “gaming”  of  the  newly  enacted  pps. 

When  we  discussed  this  comment  with  hcfa  officials,  it  appeared  to  us 
that  HCFA  did  not  agree  that  the  funding  level  had  influenced  the  scope 
of  work.  HCFA  officials  told  us  that  funding  levels  had  not  been  consid¬ 
ered  in  developing  the  scope  of  work  for  the  1986-88  contracts.  In  their 
view,  the  fact  that  the  cost  of  the  scope  of  work  had  come  out  at  about 
the  legally  required  minimum  funding  level  was  evidence  that  this  level 
was  adequate  to  meet  the  program’s  intent.  They  noted  that  the  Office 
of  Management  and  Budget  did  not  actually  approve  the  funding  level 
until  the  scope  of  work  was  available  and  indicated  that  if  hcfa  had  felt 
it  necessary  to  ask  for  additional  funding,  it  would  have  done  so. 

Our  evidence  indicates,  and  hcfa  officials  agree,  that 

the  funding  level  for  the  1986-88  scope  of  work  was  developed  and  sent 
to  the  Office  of  Management  and  Budget  for  approval  before  the  scope 
of  work  w^ls  approved, 

the  funding  level  was  at  that  time  justified  solely  on  the  basis  of  the 
legally  required  minimum  funding  level,  and 

the  funding  level  remained  at  the  minimum  after  the  development  of  the 
scope  of  work  for  the  1986-88  contracts. 

These  circumstances  indicate  to  us  that  the  minimum  legally  permitted 
funding  level  would  have  had  a  constraining  effect  on  the  development 
of  the  scope  of  work.  Regardless  of  whether  the  minimum  funding  level 
was  a  constraint  on  the  1986-88  scope  of  work,  we  believe  our  recom¬ 
mendation  is  valid  for  how  future  scopes  of  work  should  be  developed, 
and  HHS  agreed  with  this. 

The  inspector  general  said  that  hhs  also  concurred  with  our  recommen¬ 
dation  that  HCFA  collect  and  use  adequate  cost  and  performance  data  to 
set  each  pro’s  contract  funding  at  a  level  sufficient  to  provide  the  cover¬ 
age  determined  to  be  necessary  and  that  hcfa  had  taken  steps  to 
improve  the  information  base  on  which  its  estimates  are  based.  As 
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examples,  it  noted  that  pros  are  required  to  report  cost  data  under  the 
current  pro  scope  of  work  and  that  hcf’a  had  initiated  plans  to  conduct 
studies  in  several  pro  areas  to  validate  the  time  required  to  perform  var¬ 
ious  review  functions  and  the  associated  costs  of  such  efforts. 

AMPRA,  in  its  comments,  strongly  concurred  with  our  findings  on  pro 
funding,  ampra  believes  that  the  pros  have  been  underfunded  for  the 
current  round  of  contracts  and  that  hcfa  should  renegotiate  the  pro  con¬ 
tracts.  ampra  also  believes  that  hcfa  should  develop  its  future  cost  esti¬ 
mates  on  the  basis  of  actual  pro  review  costs. 
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PRO  Report  Worksheets  and  Related  Activities 


Worksheet  number 

Activity  covered  \ 

1 

Admission  Objective  1 

2 

Admission  Objective  2 

3 

Admission  Objective  3 

4 

Admission  Objective  4  (not  all  PROs  had  a  fourth  admission 
objective) 

5 

Preadmission  Review 

6 

Admission  Review 

7 

Diagnosis  Related  Group  Validation 

8 

Day  and  Cost  Outlier  Denials 

9 

Quality  Objectives  (provides  for  up  to  six  quality  objectives) 

10 

Referrals  fro.n  HCFA,  the  insoector  general,  or  Members  of  Congress 
and  others 

11 

Sanctions  and  Reconsiderations 

12 

Utilization  Problems  (covers  problems  not  found  in  the  admission 
objectives) 

13 

Quality-of-Care  Issues  (covers  quality  issues  not  found  in  quality 
objectives) 

14 

Profiling  (covers  hospital,  physician  DRG/procedure,  and  patient 
profiles) 

15,1 

Interventions  for  Admission  Objective  1 

152 

Interventions  for  Admission  Objective  2 

15.3 

Interventions  for  Admission  Objective  3 

Interventions  for  Admission  Obiective  4  (if  applicable) 

Interventions  for  Quality  Objective  1 
Interventions  for  Quality  Objective  2 
Interventions  for  Quality  Objective  3 
Interventions  for  Quality  Objective  4 
Interventions  for  Quality  Objective  5 
Interventions  for  Quality  Objective  6 
Waiver  Activity 

Intensified  Review  (covers  intensified  review  for  admissions  DRG 
validation,  and  day  and  cost  outliers) 

Private  Review 

Internal  Quality  Control  for  Review  Coordinators 
Internal  Quality  Control  for  Physician  Advisors 
Internal  Quality  Control  for  DRG  Validation  Staff 

Impact  Outside  Objectives  (covers  PRO  achievements  in  areas  not 
specifically  required  by  the  contract) 
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PRO  Evaluation  Scores,  Panel 
Recommendations,  and  Final  Decisions' 


Scores 
Section  T 


Section  1*> 

required 

Section  3** 

Panel 

Pinel 

State 

objectives 

reviews 

management 

recommendation 

decision 

Alabama 

183 

34 

208 

Unsat 

Compete 

Alaska 

Amer  Samoa/ 

359 

132 

225 

Sat 

Renew 

Guam 

99 

99 

186 

Unsat 

Compete 

Arizona" 

380 

140 

206 

Sat 

Renew 

Arkansas 

436 

157 

283 

Sat 

Renew 

California 

385 

159 

295 

Sat 

Renew 

Connecticut 

367 

140 

229 

Sat 

Renew 

Delaware 

311 

90 

199 

Unsat 

Compete 

Dist  Columoia 

269 

99 

190 

Unsat 

Compete 

Florida' 

374 

99 

240 

N/A 

Renew 

Georgia 

319 

42 

284 

Unsat 

Compete 

Hawaii 

189 

127 

107 

Unsai 

Compete 

Illinois 

343 

132 

233 

Sat 

Renew 

Indiana 

219 

142 

224 

Sal 

Renew 

Iowa 

376 

132 

275 

Sat 

Renew 

Kansas^ 

365 

116 

246 

Sal 

Compete 

Kentucky 

260 

50 

150 

Unsat 

Compete 

Louisiana 

261 

24 

239 

Unsat 

Compete 

Maine 

334 

82 

222 

Unsat 

Compete 

Maryland 

320 

116 

211 

Unsat 

Compete 

Michigan 

372 

175 

262 

Sat 

Renew 

Minnesota 

261 

51 

159 

Unsat 

Compete 

Mississippi 

394 

140 

292 

Sat 

Renew 

Missouri 

287 

99 

224 

Unsat 

Compete 

Montana 

348 

132 

234 

Sat 

Renew 

Nebraska 

231 

80 

146 

Unsat 

Compete 

Nevada" 

337 

132 

292 

Sat 

Renew 

New  Hampshire 

332 

124 

272 

Sat 

Renew 

New  Jersey 

395 

134 

248 

Sat 

Renew 

New  Mexico 

132 

17 

25 

Unsat 

Compete 

New  York 

343 

132 

291 

Sat 

Renew 

No  Carolina 

0 

132 

162 

Unsat 

Compete 

No  Dakota 

370 

132 

238 

Sat 

Renew 

Ohio 

252 

132 

175 

Unsat 

Compete 

Oklahoma 

261 

132 

214 

Unsat 

Compete 

Oregon 

227 

124 

210 

Unsat 

Compete 

Puerto  Rico' 

356 

200 

224 

Sat  Renew 

(continued) 
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Appendix  II 

PBO  EvaliuiUon  Scorra,  Panel 
Seconunendaliona.  and  Pinal  DeriHioiiti 


Score* 


State 

Saction  I" 
objectivaa 

Saction  2' 
raquirad 
reviatws 

Saction  3" 
managamant 

Panal 

racommandation 

Final 

decision 

Rhode  Island 

330 

116 

224 

Sat 

Renew 

So  Dakota 

264 

140 

181 

Unsat 

Compete 

Tennessee 

243 

159 

192 

Unsai 

Compete 

Texas 

329 

120 

237 

Sal 

Renew 

Utah 

343 

124 

175 

Unsat 

Compete 

Vermont'! 

309 

124 

210 

Unsat 

Renew 

Virgin  Islands 

0 

200 

75 

Unsat 

Compete 

Virginia 

376 

124 

268 

Sat 

Renew 

Washington 

261 

112 

103 

Unsat 

Compete 

West  Virginia 

477 

173 

276 

Sat 

Renew 

Wisconsin 

234 

99 

196 

Unsat 

Compete 

Wyoming 

351 

124 

269 

Sat 

Renew 

^Idaho  Massachusetts  Pennsylvania  and  South  Carolina  were  not  evaiualed  due  to  termination  oi  the 
contract  Colorado  scores  were  not  avaiiaCie  Some  scores  have  been  rounded  to  remove  decimals 

'’Section  I  Total'500  Minimally  Met  326  Fully  Met  38S  Exceeded  Mel  446 

'Section  II  Total  175  Minimally  Mel  116  Fully  Met  176 

'’Section  III  Total  325  Minimally  Mel  224  Fully  Met  318 

'Panel  recommendation  lor  renewal  overruled  by  HCFA  headquarters  officials  but  PRO  successfully 
appealed  decision 

'Actual  panel  recommendation  unclear  from  available  documentation 
JPanel  recommendation  overruled  by  HCFA  headquarters  officials 
"Contract  competed  due  to  inability  to  negotiate  price 
'Non  PPS  area  with  atypical  PRO  contract 
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Comments  From  the  Department  of  Health  and 
Human  Services 


SfP  4  198? 


Mr .  Richard  L.  Foqel 
Assistant  Comptroller  General 
U.S.  General  Accounting  Office 
Washington,  D.C.  20548 

Dear  Mr.  Foqel: 

The  Secretary  asked  that  I  respond  to  your  request  for  the 
Department's  comments  on  your  draft  report,  "Medicare:  Better 
Controls  Needed  for  Peer  Review  Organizations'  Evaluations."  The 
enclosed  comments  represent  the  tentative  positici  of  the 
Department  and  are  subject  to  reevaluation  when  the  final  version 
of  this  report  is  received. 

We  appreciate  the  opportunity  to  comment  on  this  draft  report 
before  its  publication. 

Sincerely  yours. 


Vi  tt  . 


Richard  P.  Kisserow 
Inspector  General 
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('onunrnlK  Pritiii  thr  Depart nM‘nl  i>f  Health 
and  Human  Servireo 


OcMgHtB  erf  the  Dyut—tt  erf  ^M^th  and  tt—n  SarviocB 
cm  tte  flaaeraf  Aooountinii  Office  Etaft  Beport, 
Hettag  Oadjola  iinwiWl  In  Baer  Itevieir 
Omani  aatiOBB'  BtmllMtiiCB^ 


Overview 


GWD's  report  was  prepared  at  the  request  of  the  Chairman  and  Rankinq 
Minority  Member  of  the  Subcaimittee  on  Health,  Senate  Ccmnittae  on 
Finetnce.  Ttvey  etsked  that  GWD  assess  (1)  the  Health  Care  Financing 
IVininistration ' s  (HCFA's)  Peer  Review  Organization's  (PWO's)  evaluation 
methodology  and  (2)  the  adequacy  of  HCFA's  routine  monitoring  of  the  PBOs 
for  identifying  performanoe  problems.  In  ctddition,  GftD  was  asked  to 
determine  how  HCFA  decided  on  the  funding  level  for  the  new  contracts. 

According  to  GAD,  the  evaluation  process  involved  instructions  that  were 
incxnsistent,  incorrect  or  not  properly  inplemented  by  the  evaluation 
panels  and  docimentation  of  the  panels'  and  HCFA's  decisions  was  not 
always  adequate.  Additionally,  GAD  reports  that  the  substantial  portion 
of  PROS  that  failed  the  evaluations  as  well  as  HCFA's  lack  of  awareness  of 
the  extent  of  the  problems  identified  through  the  evaluation  process 
showed  that  HCFA's  routine  monitoring  during  the  oontract  period  had  not 
identified  and  corrected  PRD  performance  problems. 

We  would  note  that  GAO's  review  covered  HCFA's  monitoring  and  evaluation 
efforts  for  the  PRO'S  first  contracting  cycle.  Ftor  the  first  contracting 
cycle,  HCFA  developed  a  process  for  ongoing  monitoring  which  utilized  a 
Peer  Review  0rg2inizaticn  Monitoring  Protocol  emd  Tracking  System  (PROMPTS) 
document  as  a  means  of  obtaining  cxnsistency  on  a  national  basis  of  the 
monitoring  of  the  54  individual  PROS.  The  purpose  of  the  final  evaluation 
was  to  determine  whether  a  PRO'S  performanoe  was  sufficient  to  merit  a 
noncxxrpetitive  renrnml  of  its  oontract.  Those  ETOs  that  did  not  meet 
oontract  requirements  during  the  contract  period  were  identified  emd 
terminated.  During  this  first  contracting  cycle  this  combined,  ongoing 
process  resulted  in  three  PROs  being  terminated  and  a  final  determination 
that  25  PROS  did  not  perform  at  a  high  enough  level  to  warramt 
nonodipetitive  rental. 

It  is  noteworthy  that  GAO's  ocrprehensive  reviaa  of  this  process  did  not 
reveal  a  single  instance  where  a  flaw  in  the  process  resulted  in  an 
inappropriate  decision  to  offer  or  withhold  a  noooonpetitive  contract 
renewal.  Ttiis  tends  to  confirm  the  ultimate  success  of  HCFA's  monitoring 
and  evaluation  programs. 


GAD  Reconwendation 

The  Secretary,  HHS,  should  instrurt  the  Adninistrator  of  HCFA  to  assure 
that  in  future  PRO  evaluations,  the  evaluation  process  hats  sufficiait 
internal  controls  to  assure  that  evaluations  aure  consistently  allied  and 
that  decisions  resulting  from  the  evaluation  aure  atdequately  dkxumented.' 
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Pac>e  ^ 


Department  Conment 

Me  cxncur  with  this  recunnef idation  and  steps  fave  been  taken  to  uiprove 
the  internal  controls  that  were  in  plcioe  durinq  the  first  contract inq 
cycle.  For  exanple,  as  part  of  the  evaliiation  process,  suimary  sheets  are 
prepared  by  a  review  panel  ccnprised  of  central  and  regional  office 
personnel  to  ensure  that  PSD  evaluations  are  ccnplete  and  accurate.  Sli>,. . 
sumary  sheets  will  be  controlled  to  (iocunent  the  nature  of  the 
decision-making  process.  Internal  controls  will  continue  to  be  inproved 
on  an  ongoing  basis. 

GUO'S  oonoem  in  this  ^u'ea  results  frcm  findings:  that  HCFA  lacked 
internal  controls  over  the  evaluation  process;  that  there  were 
inconsistencies  in  the  instructions  for  evaluation  euid  profiling;  that 
panel  instructions  were  incorrect  ami  that  evaluation  results,  in  some 
instances,  were  not  adequately  docunentad.  There  were  two  ca5>es  nhere  the 
decision  did  not  e^pear  to  be  fully  docimented  in  the  file.  In  both  of 
these  instances  the  FHO  in  question  was  nonocnpetitively  renewed.  These 
two  cases  did,  in  f^K:t,  slip  through  the  controls  in  place  at  that  time. 
While  this  was  inappropriate,  it  should  be  noted  that  due  to  the  nimber  of 
evaluations  being  done  simultaneously,  we  put  emphasis  on  contracts 
proposed  for  nonrenewal  as  virtually  all  of  these  decisions  were 
appealed.  As  stated  above,  HCFA  hM  taken  steps  to  improve  the  evaluation 
internal  controls  based  on  its  esqierienoe  during  the  first  PRO  contracting 
cycle. 


GAO  Reocmwendation 

That  tte  Secretary  of  HHS  direct  the  Administrator  of  HCFA  to  provide 
criteria  to  en^J3le  HCFA  p^sonnel  to  differ^tiate  between  accS^t^le  and 
unacceptable  performance  in  the  routine  monitoririg  of  PROs'  activities. 

Department  Ocnwent 

We  generally  ^lgree  with  this  recaimendation  and  will,  where  eppropriate, 
continue  to  provide  criteria  to  HCFA  personnel  for  differentiating  between 
acToeptable  and  unacceptable  performance. 

This  reoamnendatian  resulted  from  GAD's  ooncem  that  the  ongoing  PRO 
monitoring  protocol  (PRCMPTS-2)  does  not  contain  adequate  criteria  to 
enable  HCFA  monitoring  personnel  to  differentiate  between  acceptable  and 
unacceptable  performance.  The  criteria  to  be  used  by  regional  office  (?(0) 
monitoring  personnel  (i.e.,  piroject  officers)  have  been  given  to  RD 
personnel  outside  of  the  PRCMPTS-2  docunent.  The  decision  to  do  so  is 
based  on  the  premise  that  PRCMPTS-2  was  never  intended  to  function  as  an 
instructional  device  or  manual  for  pioject  officers.  Training  programs 
have  been  held  for  regicraU.  project  officers  and  medical  review  personnel, 
and  instructional  materials  have  been  provided.  HCFA  believes  criteria 
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oontairaed  in  the  ir»iividual  PW'  cxxitracts,  PRO  instructional  issuances  and 
the  PRO  moritorinq  protocols  as  well  as  training  cind  instructional 
materials  specific  to  F1O0*rS  will  result  in  appropriate  monitoring  of  PP^ 
(lerformanoe . 

For  the  finding  applicable  to  this  recaimendation,  GAD  also  criticized 
HCPA  during  the  first  PRO  cjontracting  cycle  for  not  detecting  auid 
correcting  many  instances  of  poor  PRO  perforronoe  before  the  cxmtract 
renewal  evaluation.  HCFA  believes  that  the  monitoring  docunent  did  enzd>le 
It  to  identify  poor  performers.  This  is  evident  by  the  three  PRO  oontract 
terminations  prior  to  the  evaluation.  The  evaluation  further  identified 
probleBB  that  kept  a  PRO  frcm  achieving  a  level  of  perfomance  sufficient 
for  nonccnpetitive  ren««il,  but  not  poor  enough  for  termination. 

GIWD  acknowledges  that  the  PRCMPTS-2  monitoring  document  alleviates  rany  of 
Its  conoems.  These  inprovaaents  were  made  as  a  result  of  HCFA's 
experienop  during  the  first  PRO  contracting  cycle.  HCFA  v«rill  continue  to 
make  uiprovements  as  experience  dictates. 


GAO  Recomnendation 


That  the  Seer 


of  HHS  direct  the  Attninistrator  of  HCFA  to: 


--determine  the  scope  of  review  needed  to 


am's  intent  and  use  this  as  the  start 


and 


telv  meet  the 


int  for 


We  concur  with  this  reccwnendation  and  will  continue  efforts  in  this  area. 

This  rajaiina riat ion  apparently  resulted  frem  GAD'S  misunderstanding  cis  to 
how  the  scope  of  review  was  developed.  The  scope  of  work  supporting  PRO 
contracts  was  developed  based  upon  experience  gained  frem  medical  review 
by  Professional  Standards  Reviaif  Organizations  and  fiscal  intermediaries 
aixi  was  designed  to  address  "gaming"  of  the  nawly  enacted  prospective 
payment  systan  (PPS).  We  therefore  focused  PRO  review  on  identifying 
and/or  preventing  unnecessary  ^ldnissions.  Specific  review  requirements 
were  established  to  monitor  areas  most  susceptible  to  "gaming"  such  as 
over-utilization,  erroneous  diagnosis  related  group  classification, 
transfers,  rescissions  and  outliers.  Focusing  requiranents  were  also 
included  to  ensure  that  PRDs  intensified  review  in  identified  problan 
areas . 

As  we  gained  experiencje  with  PPS  and  PRO  re^hew,  we  began  during  the  first 
scxipe  of  work  to  strengthen  PRO  activities  in  the  area  of  quality  of  care 
assureuice  and  to  specifically  monitor  PRO  performance  in  that  area  to 
ensure  that  PRDs  identified,  assessed,  and  corrected  (quality  problems. 
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In  addition,  the  second  scope  of  vicirk  provided  for  an  even  more  intensive 
review  process  (requiring  quality  review  of  each  case,  adding  ireindated 
quality  screens,  using  HCFA-identif ied  potential  quality  of  care  problems 
to  develop  objectives,  and  requiring  greater  physician  participation).  As 
with  the  first  scope  of  work,  it  contained  provisions  for  focusing  review 
on  problems  identified. 

i 

GW3  Reocrinendation 

— collect  and  use  adequate  cost  c>nd  perfornance  data  to  set  each 

pro's  contract  funding  at  a  level  sufficient  to  provide  the 

coverage  determined  to  be  necessciry.  | 

Department  Caiment 

We  concur  with  the  need  for  adequate  cost  and  performance  data  to  set  each 
pro's  contract  funding  level  euid  will  cuntinue  to  inprove  the  base  upon 
which  its  estinabes  are  predicated.  Under  the  current  PRO  scope  of  work, 

PROS  eure  required  to  report  cost  data.  In  ciddition,  HCFA  has  initiated 
plans  to  conduct  engineering  studies  in  several  PRO  areas  which  will 
validate  the  time  required  to  perform  various  review  functions  eund  the 
associated  oost  of  such  efforts.  This  information  should  assist  us  in 
establishing  individual  PRO  funding  levels  and  negotiating  PRO  contraKits 
for  the  third  scope  of  work. 

This  reccnmendation  resulted  from  the  finding  that  HCFA  cost  estimates 
were  based  on  a  model  v^ch  reflected  significant  uncertainties  (percent 
of  cases  revi««d  amd  oost  per  case).  The  national  funding  model  utilized 
to  develop  the  Goverrinent  estinates  was  designed  to  serve  as  a  tool  (not  a 
hard  auxj  fast  rule)  for  negotiators  to  evaluate  each  PRO'S  oost  proposal 
auid  maite  aKljustments  for  justifiad>le  differences. 

As  GAO  stated,  HCFA's  funding  model  was  bamed  on  data  from  HCFA's  actuaury, 
the  Depeutment  of  Labor,  professionaU.  medical  aissociations  aind  the 
individual  PTOs.  It  also  incorporated  professional  estimates  of  the  time 
elements  needed  to  perform  required  review  functions. 

I 

HCFA  found  the  funding  model  extremely  valuad>le  in  negotiating  PRO 
contracts.  The  conclusion  reached  ty  GAO  that  the  overall  average  cost 
contracted  per  review  was  less  tham  2  percent  above  the  cost  estimated  by 
the  HCFA  nodel  is  evidence  that  the  HCFA  model  was  indeed  effective  and 
appropriate.  As  enphasized  above,  HCFA  is  continuing  to  seek  ways  to 
inprove  the  model  upon  which  PRO  estimates  aure  based. 
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Review  Association 


American  Medical  Peer  Review  Association 


440  FIRST  STREET,  NW 


•  SUITE  510 


•  WASHINGTON .  D.C  20001  • 


(202I  628-1853 


August  30,  1987 


Richard  L.  Fogel 
Assistant  Comptroller  General 
General  Accounting  Office 
Human  Resources  Division 
Washington,  D,C.  20548 


Dear  Mr.  Fogel: 


The  American  Medical  Peer  Review  Association  (AMPRA)  representing 
peer  review  organizations  (PROs)  is  appreciative  of  the  opportunity 
to  comment  on  GAO's  report  entitled,  "Better  Controls  Needed  for  Peer 
Review  Organizations'  Evaluations". 


The  GAO  is  to  be  conplimented  for  conducting  a  thorough  and  well 
documented  study  of  the  evaluation  of  PROs.  Over  the  last  years, 
AMPRA  has  expressed  concerns  for  the  lack  of  consistency  and 
objectivity  in  the  PRO  evaluation  process.  AMPRA  is  grateful  that 
our  analysis  is  now  confirmed  by  the  GAO  report.  Our  association  is 
in  strong  agreement  with  the  many  report  findings  and  endorses  the 
report  reconmiendations.  These  recornmendations,  if  adopted  by  HCFA, 
should  improve  the  evaluation  of  PROs  in  the  future. 


AMPRA  would  like  to  make  the  following  comments  and  recommendations: 

1.  We  agree  that  deficiencies  in  PRO  operations  should  be  pointed 
out  to  the  PRO  as  early  in  the  contract  cycle  as  possible. 
Waiting  for  the  final  evaluation  before  deficiencies  are 
uncovered  is  not  fair  to  the  PRO  that  has  assumed  HCFA  is 
satisfied  with  operations  until  that  point.  In  addition,  it 
does  not  permit  time  for  corrective  action. 

2.  We  agree  that  PROMPTS-2  is  an  improvement  over  earlier 
evaluation  instruments.  We  strongly  share  GAO's  concern, 
however,  that  PR(JMPTS-2  lacks  specific  criteria  to  measure 
acceptable  or  unacceptable  performance.  Without  defined 
criteria,  the  subjective  judgements  of  different  HCFA 
personnel  will  continue  to  influence  final  evaluations  of 
PROS.  AMPRA  strongly  endorses  the  reconmendat i on  that 
specific  and  objective  evaluation  criteria  be  developed. 


1 

1 


I 
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August  30,  1987 
Mr.  Fogel 


3.  AMH3A  strongly  recommends  that  each  PRO,  as  part  of  the 
evaluation  process,  be  able  to  examine  and  rebut  the 
"independent  analysis"  prepared  by  HCFA  regional  office 
personnel.  Further,  final  evaluation  panel  findings, 
recommendations  and  numerical  scores  should  be  shared  with 
every  PRO.  For  PROs  that  are  non-renewed,  it  is  particularly 
important  that  all  evaluation  materials  be  shared  so  a 
specific  rebuttal  to  all  identified  deficiencies  can  be 
possible.  Non-renewed  PROs  received  a  generalized  summary  of 
deficiencies  in  the  form  of  a  letter  when  only  a  complete  set 
of  evaluation  materials  will  permit  a  thorough  response.  For 
renewed  PROs,  the  complete  set  of  evaluation  materials  can 
only  assist  in  identifying  weaknesses  in  PRO  operations  that 
need  improvement. 

4.  AMPRA  remains  perplexed  by  the  relationship  of  SuperPRO  to  the 
PRO  evaluations.  It  appears  that  SuperPRO  had  little  or  no 
influence  on  final  determinations.  Furthermore,  it  has  lieen 
AMPRA's  observation  that  the  SuperPRO  efforts  to  monitor 
medical  review  decisions  by  PROs  duplicates  regional  office 
medical  review  team  oversight .  AMPRA  recomnends  that  the  role 
of  the  SuperPRO  be  clarified  and  that  a  single  entity  be  given 
responsibility  to  monitor  PRO  medical  review  determinations. 

5.  AMPRA  strongly  concurs  with  the  GAO  report  findings  on  FRO 
funding.  It  has  been  obvious  that  HCFA  has  consistently 
funded  the  PRO  program  at  the  legal  minimum  recfuired  by  law. 
AMPRA  believes  that  this  minimum  level  is  grossly  inadequate 
to  fund  labor  intensive  quality  reviews  and  new 
congressionally  mandated  review  activities.  Furthermore,  we 
agree  that  "HCFA  used  several  assumptions  to  develop  its 
initial  cost  estimates  for  PRO  contracts  which  result  in  a 
degree  of  uncertainty  about  these  estimates".  AMPRA  believes 
that  HCFA  significantly  underestimated  the  time  PRO  reviewers 
would  take  to  review  cases  in  the  second  round  of  PRO 
contracts.  An  AMPRA  survey  shows  that  review  time  has  more 
than  doubled  and  that  both  physician  referrals  and  the  time  it 
takes  a  physician  to  review  cases  has  jumped  significantly. 
This  increased  review  "intensity"  was  not  offset  by  the 
decrease  in  volume  of  review  in  the  second  round  of 
contracts.  For  nearly  every  PRO,  review  expenses  exceed 
review  costs. 

AMPRA  recomnends  that  HCFA  renegotiate  the  base  contracts  in 
recognition  of  the  fact  that  their  cost  estimates  and 
assumptions  were  "uncertain"  and  now  proved  wrong  by  actual 
PRO  experience.  For  the  future,  AMPRA  recomnends  that  HCFA 
cost  estimates  be  developed  in  recognition  of  actual  PRO 
review  costs  and  shared  with  the  PFiO  comnunity. 
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Page  Three 
August  30,  1987 
Mr.  Fogel 


Once  again,  AMPRA  appreciates  the  opportunity  to  conment  on  GAO's 
report.  We  sincerely  hope  that  report  reconmendations  will  be 
implemented. 


Sincerely, 


Andrew  H.  Webber 
Executive  Vice  President 
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